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The  most  significant  problem  facinq  the  computer 
profession  tolay  is  manifested  in  two  major  complaints  about 
software:  it  is  too  expensive  and  unreliable.  Most  comput- 

er professionals  recognise  the  high  cost  as  larqaly  a symp- 
tom of  the  latter  complaint.  The  high  incidence  of  errors 
in  software  is  the  underlying  reason,  for  unreliability. 

The  number  of  errors  uncovered  during  the  software  life 
cycle  a as  a significant  impact  upon  the  cost  in  terms  of  re- 
sources (personnel  and  computer)  needed  to  correct  the  er- 
rors. The  correction  cost  is  a function  of  when,  in  the 
software  life  cycle,  an  error  is  found.  Software  errors 
found  during  the  development  phase  generally  cost  less  to 
correct  than  errors  which  occur  during  the  operations  phase. 
Therefore,  it  is  necessary  to  detect  software  (program)  er- 
rors as  early  as  possible-- preferably  before  the  software  is 

iv 


made  operational.  Also,  it  is  necessary  to  predict  the  num- 
ber of  residual  errors  in  a program  to  determine  if  anl  when 
the  proqram  goes  operational. 

Proqraa  structural  characteristics  metrics  (internal 
complexity)  is  a means  of  estimating  the  number  of  errors. 
Thirteen  unrelated  characteristics  metrics  are  used  to  de- 
fine 7 local  complexities — control  flow,  input/output,  data 
handlinq,  computational  structural  design,  interface,  ana 
data  use — which  are  predictors  of  the  number  of  errors  in 
COBOL  proqraras.  Linear  models  for  each  of  these  metrics  are 
available  for  predicting  software  errors. 

Models  developed  from  these  metrics  can  be  used  to  pre- 
dict the  number  of  errors  in  COBOL  proqrums.  The  "pest" 
sinqle  variable  model  for  predicting  errors  is  the  Control 
Plow  Complexity  metric  model.  The  "best"  multiple  variaole 
model  for  predicting  errors  is  the  cne  that  contains  all  7 
local  complexity  metrics.  The  latter  model  can  be  used  wnen 
dealing  with  many  types  of  programs  that  are  developed  by 
different  organizations.  However,  each  organization  snould 
estimate  the  model  parameters  relative  to  error  data  from 
its  development  proiects. 
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INTRODUCTION 


The  primacy  consideration  in  any  system  is  that  it  per- 
forms properly  whenever  the  user  wants  to  use  it.  For  com- 
puter systems,  consisting  of  hardware,  software  and  man- 
machine  interfaces,  the  most  widely  accepted  and  meaningful 
measure  of  performance  is  total  system  reliability.  Total 
system  reliability  is  defined  as  the  probability  that  every 
subsystem  performs  as  intended  for  the  necessary  time  and 
under  the  conditions  of  customer  use.  Thus,  there  is  an  ob- 
vious need  to  measure  the  reliability  of  the  software  sub- 
system as  well  as  the  hardware  and  the  man-machine  subsys- 
tems. But  the  most  significant  problem  facing  the  computer 
profession  today  is  a software  problem  that  is  manifested  in 
two  major  complaints:  software  is  too  expensive  and  soft- 
ware is  unreliable.  Most  software  professionals  recognize 
the  former  problem  as  largely  a symptom  of  the  latter.  Al- 
though this  paper's  main  focus  is  primarily  on  the  proDlem 
of  unreliaple  software,  the  problem  of  high  cost  is  an 
indirect  issue  because  of  its  relationship  to  unrei iaoi lit y . 
Therefore,  to  put  the  problem  of  unreliability  into  proper 
perspective,  the  problem  of  high  cost  will  be  discussed 
briefly.  This  discussion  will  stress  the  important  role 
that  reliability  plays  in  increasing  the  cost. 
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The  riigh  Cost  of  Software 

There  are  several  reasons  why  software  is  so  expensive. 
The  rest  of  this  section  will  discuss  a few  of  them. 

Computer  systems  are  becoming  more  complex  as  faster 
and  more  versatile  hardware  evolves.  The  resultant  sophis- 
ticated uses  of  the  computer  systems  demand  that  programmers 
develop  reliable  software  to  drive  the  computer  system. 
Additionally,  the  man-machine  interface  which  is  generally 
handled  by  software  is  becoming  more  and  more  sophisticated. 
Consequently,  software  is  becoming  more  and  more  complex 
because  of  the  hardware  and  the  man-machine  interface  sub- 
systems. This  increases  software  development  cost. 

As  we  continue  to  automate  processes  which  control  our 
life-style — bank  accounts,  air  traffic  control,  medical  sys- 
tems, and  defense  systems — we  have  to  trust  more  and  more  in 
the  reliable  functioning  of  software.  Nowhere  is  this  more 
evident  than  in  the  military  where  computers  are  Deing  used 
increasingly  as  the  heart  cf  sophisticated  weapon  systems 
such  as  the  B- 1 bomber  or  a real-time  command  and  control 
system.  They  control  their  environments  by  receiving  data, 
processing  it  and  returning  results  fast  enouqh  to  affect 
the  functioning  of  their  environments.  Reliable  functioning 
of  software  is  also  critical  in  an  on-line  banking  system 
where  a software  error  (failure)  may  result  in  a loss  of 
thousands  of  dollars.  To  develop  reliable  software,  we 
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spend  lore  and  more  resources  on  quality  control  during 
software  development,  thus,  increasing  the  cost  directly. 

Software  is  a Diq  business  in  the  U.  S.  today.  Tne 
annual  cost  of  software  is  approximately  20  billion  dollars. 
Its  rate  of  qrowth  is  qreater  than  that  of  the  economy  in 
qeneral.  Compared  to  the  cost  of  hardware,  the  cost  of 
software — development  and  maintenance — is  escalating  along 
the  lines  in  Fiqure  1(1],  Studies  f 2 , 3 1 indicate  that 
software  demand  over  the  years  1975-1585  will  qrow  about 
21-23  percent  per  year.  This  is  considerably  faster  than 
the  qrowth  rate  in  software  supply  at  the  current  estimated 
qrowth  rates  of  the  labor  force  and  its  productivity  per  in- 
dividual which  has  a combined  qrowth  rate  of  aoout  11.3-17 
percent.  Because  of  the  demand  and  a shortage  of  experi- 
enced programmers,  error  prone  software  will  be  developed. 
Poor  software  reliaoility  will  be  revealed  by  an  excessive 
number  of  software  errors  resulting  in  higher  maintenance 
cost  and  customer  dissatisfaction. 

Errors  discovered  after  the  software  is  operational 
will  impact  greatly  upon  the  cost  of  software  because  of 
computer  resources  and  manpower  needed  to  correct  tne  er- 
rors. About  70  percent  of  today's  software  dollar  goes  into 
software  maintenance,  and  this  number  will  likely  qrow  [4], 
This  percentage  varies  by  organization.  Maintenance  cost 


versus  development  cost  for  different  organizations  is 
depicted  in  Fiqure  2 f 5 1.  DeRose  [6]  estimates  that  it 
costs  the  Department  of  Defense  J75  an  instruction  to  devei- 
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op  aviation  software,  but  that  the  maintenance  cost  is  a lot 
more.  On  one  particular  aircraft  computer,  maintenance  cost 
ran  as  hiqh  as  $4000/instruction  f 7 ]. 

Proposed  solutions  to  the  cost  problem  invariaoly  in- 
volve an  attempt  to  raise  programmer  productivity  by  de- 
visinq  tools  and  techniques  to  allow  programmers  to  woric 
more  quickly.  But  it  should  be  obvious  that  tne  high  cost 
of  software  is  largely  due  to  reliaoility  problems.  Cost  is 
not  usually  lowered  significantly  by  increasing  programmer 
productivity  if  the  latter  is  a measure  of  the  speed  of 
desiqninq  and  coding  the  proqram.  Depending  on  the  situa- 
tion, attempts  to  increase  programmer  productivity  can  in- 
crease cost.  The  best  way  to  sharply  decrease  software  cost 
is  to  reduce  maintenance  and  testing  cost  by  devising 
techniques  to  produce  reliable  software.  This  is  the  pri- 
mary motivation  for  a software  reliability  theory.  The  next 
section  will  discuss  the  software  reliability  problem. 


The  Software  Reliability  Problem 


definition 


Software  reliability  is  the  probability  that  a given 
proqram  operates  correctly,  without  an  error,  tor  some  time 
period  on  the  machine  for  which  it  was  designed.  Correctly 
means  that  the  proqram  performs  as  the  ultimate  user  wants 
it  to. 
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The  Problem 

The  hiqh  incidence  of  errors  in  software  is  the 
underlvinq  problem  of  software  reliaoility.  It  would  ae 
fortunate  if  the  well-developed  theory  of  hardware  reliabil- 
ity (see  Appendix  A)  could  be  used  to  predict  or  enhance  tae 
reliability  of  software.  Unfortunately,  this  is  not  tae 
case  since  hardware  reliaoility  theory  is  based  mainly  upon 
the  statistical  analysis  or  random  and  wear-out  failures  of 

t 

components  with  aqe.  In  contrast  software  is  not  subject  to 
wearout  failures  once  it  is  debuqqed. 

There  are  other  important  differences  Detween  hardware 
and  software  which  make  the  hardware  reliability  tecnniques 
difficult  to  apply  to  software.  The  elementary  components 
of  software  are  instructions.  They  do  not  wear,  orealt,  or 
deteriorate.  All  software  errors  are  in  some  sense  desiqn 
or  implementation  errors  [3]  which  are  comparable  to  Darn-in 
errors  in  hardware.  When  errors  are  found,  they  can  be  cor- 
rected and  are  no  lonqer  present  in  the  proqram.  In  gener- 
al, proqrams  are  more  complex  than  corresponding  hardware 
loqic.  Larqe  proqrams  are  profcaoly  the  most  complex  oojects 
Duilt  by  man.  Some  of  them  have  millions  of  instructions. 
The  complexity  of  these  proqrams  is  so  qreat  that  it  is  not 
well  understood  what  the  proqram  can  or  can  not  do.  Final- 
ly, there  is  a lack  of  a scientific  basis  for  under standinq 
the  nature  of  proqrams.  In  contrast,  the  scientific  basis 
of  most  hardware  elements  is  well  known. 
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fur^ose 

Fiqure  3 shows  a summary  of  current  experience  on  the 
relative  cost  of  correctiuq  software  errors  as  a runction  of 
the  software  life  cycle  phase  in  waich  taey  are  corrected 
f 5 1.  For  obvious  reasons,  it  is  desirable  to  predict  tae 
number  of  errors  in  a software  systea  at  the  earliest  moment 
in  the  software  life  cycle  (development  and  operational 
phases).  Unfortunately  there  is  no  proven  technique  in 
practice  today. 

The  research  done  to  date  suqqests  the  hypothesis  that 
profiles  of  actual  proqrara  characteristics  (internal  com- 
plexity) are  qood  predictors  of  the  numoer  of  errors  iu  a 
proqraa.  This  paper  will  present  the  results  of  an  analysis 
of  error  data  to  determine  if  actual  proqram  characteristics 
are  predictors  of  the  number  of  errors,  propose  a model  for 
predictinq  the  number  of  errors  in  C030L  proqrams,  and  dis- 
cuss the  application  of  this  model  to  software  reliability. 
The  rest  of  this  Chapter  and  Chapter  II  and  III  contain 
backqround  information  only.  The  reader  is  directed  to 
continue  readinq  this  report  at  Chapter  IV  if  he  is  already 
familiar  with  software  enqineerinq  and  software  reliaoility 
concepts. 


Surrey  of  Belated  Research 

Over  the  past  ten  years,  several  investiqat ions  in  the 
area  of  software  reliability  and  phenomenoloq y have  been 
undertaken.  As  a result  of  these  investiqa ticns,  reliaoili- 
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ty  models  which  attempt  to  descrioe  the  failure  of  software 
have  been  proposed  and  discussed.  These  models  were  derived 
from  hardware  reliability  and  have  not  been  very  successful 
. This  failure  is  primarily  founded  on  two  reasons. 

One,  there  are  fundamental  differences  between  software 
phenomenology  and  the  hardware-oriented  assumptions  on  which 
the  models  were  based.  The  failure  mechanism  of  a hardware 
component  is  by  chance  or  by  component  wear-out  whereas  tne 
failure  mechanism  of  a program  is  a function  of  tne  number 
of  remaining  errors  in  the  program. 

Two,  the  fundamental  statistical  issues  which  emanate 
from  the  use  of  these  models  have,  by  and  large,  been 


ignored.  These  issues  pertain  to  model  verification,  tae 
development  of  a procedure  which  formalizes  the  testing  and 

| 

debugging  of  software,  and  parameter  estimation.  In  partic- 
, ular,  the  success  of  the  models  depends  largely  on  tne 

estimation  of  the  original  number  (N)  of  errors  in  software 
and  the  constant  of  proportionality  (K)  used  in  determining 
failure  rate.  Of  the  several  methods  used,  the  method  of 
maximum  likelihood  gives  the  most  reasonable  estimators  for 
. N and  K f 3-101.  However,  this  method  does  not  yield  satis- 

factory results  r 1 0 1. 

Models  are  being  developed  which  explain  previous  error 
histories  in  terms  of  appropriate  program  phenomenology. 
These  models  are  based  on  a view  of  a program  as  a maoping 

1 
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input  space,  distributed  according  to  an  operational  pro- 
file; and  of  testing  as  a sample  of  points  from  the  input 
space,  r 11,121  (see  Figure  4).  This  approach  can  ae  used 
conceptually  as  a means  of  appropriately  conditioning  time- 
driven  reliability  models  [5].  But,  «e  still  are  not  able 
to  truly  estimate  the  number  of  errors  in  software. 

Additional  insights  into  reliability  estimation  have 
come  from  analyzing  the  software  errors  relative  to  actual 
characteristics  of  programs.  Currently,  it  seems  that  a 
measure  of  proqram  complexity  offers  the  best  estimator  for 
the  number  of  residual  errors  in  a proqram.  Akiyama  [ 13] 
concludes  that  the  number  of  proqram  errors  is  stronqly  cor- 
related to  the  number  of  conditions  plus  the  number  of  calls 
to  other  programs  rather  than  proqram  size.  Lipow  and 
Thayer  I"  14]  suqqests  the  interesting  hypothesis  tnat  the 
number  of  proqram  errors  can  be  best  predicted  by  a measure 
of  the  internal  complexity  of  proqrams.  They,  using  empiri- 
cal data,  concluded  that  the  number  of  software  errors  found 
in  proqrams  written  in  JOVIAL  could  be  predicted  ny  tae  num- 
ber of  branches,  a measure  of  proqram  internal  complexity. 
Herndon  and  Lane  [15]  developed  an  approach  to  the  quantifi- 
cation of  software  errors  as  a function  of  module  complexi- 
ty. Module  complexity  is  based  upon  module  composition. 

The  complexity  measure  was  shown  to  be  a useful  managerial 
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tool.  Proqram  components  with  hiqh  complexity  indicators 
should  receive  more  attention  than  cnes  with  low  complexity 
indicators. 


There  have  been  several  other  in vestiqations  into  pro- 
qram complexity  that  did  not  address  the  error  problem. 

These  are  briefly  summarized  below.  Flynn  fib]  suqqests 
that  the  number  of  nodes  in  the  smallest  pa th-isomor pnic 
proqram  scheme  may  be  a useful  measure  of  inherent  proqram 
complexity.  Sullivan  f17]  proposes  several  complexity 
measures — cl,  c2,  c3,  pi  and  p2-  The  cl,  c2  and  c3  measures 
deal  with  control  flow  qraphs  of  prcqrams.  The  pi  and  p2 
measures  deal  with  data  flow  graphs  of  proqrams.  This  re- 
port aasically  concludes  that  the  number  of  conditions  plus 
1 is  a complexity  measure  of  the  control  flow  of  a program. 
ScCabe  r 181  develops  a qraph- theore tic  complexity  measure — 
the  number  of  conditions  in  a proqram  plus  1.  He  illus- 
trates how  it  can  be  used  to  manage  and  control  proqram  com- 
plexity. Additionally,  he  proves  that  complexity  is  inde- 
pendent of  proqram  size.  It  is  appropriate  at  this  point  to 
stress  that  most  all  of  the  software  reliability  models 
employ  the  proqram  size.  This  may  be  one  of  the  reasons  why 
the  models  have  not  been  very  successful  in  modeling  the 
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failure  rate  of  proqrams 


cioline  involvinq  a multiplicity  of  specialized  orancnes — 
Requirements  Enqineerinq,  Theory  of  Proqram  Structures,  Pro- 
qramminq  flethodoloqy,  Software  Reliability,  Software  Project 
Hanaqeaent,  etc.  This  chapter  will  briefly  discuss  those 
concepts  relative  to  estimatinq  the  number  of  residual  er- 
rors in  proqrams. 

It  is  perhaps  best  to  view  this  chapter  as  an  attempt 
to  identify  the  underlyinq  concepts  of  software  enqineerinq 
in  a fora  that  permits  the  main  issues  of  this  paper  to  be 
better  understood. 


Definitions 

Software  includes  not  only  computer  proqrams,  out  also 
the  associated  documentation  required  to  develop,  operate, 
and  maintain  proqrams.  The  qeneration  of  timely  documenta- 
tion is  an  inteqral  part  of  the  software  development  process 
f5,251. 
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Software  Sngineering  is  the  practical  application  of 
scientific  knowledge  in  the  design  and  construction  of  com- 
puter programs  and  the  associated  documentation  required  to 
develop,  operate,  and  maintain  them.  This  definition  covers 
the  entire  software  life  cycle  (see  Figure  5),  thus  includ- 
ing redesiqn  and  modification  activities  which  are  often 
called  "software  maintenance"  f 5 ]. 

The  Goals  of  Software  Engineering 

There  are  four  fundamental  goals  of  software  engineer- 
ing: modifiability,  efficiency,  reliability,  and  understand- 
aoility  f 26  ].  Boehm  f 27  1 provides  a larger  list  which  he 
calls  characteristics  of  software  quality  (Figure  6).  In 
what  follows,  this  paper  addresses  some  of  these  important 
qoals,  those  considered  basic  in  nature. 

Modifiability  implies  controlled  changes  in  which  some 
parts  are  unchanged  while  others  are  altered,  all  in  such  a 
way  that  a desired  result  is  obtained.  Modifiability  is 
difficult  to  achieve  because  changes  occur  for  many  reasons. 
For  example,  when  transferring  software  to  a new  computer 
or  operating  system,  it  is  desirable  to  keep  invariant  the 
logical  effects  of  the  system,  limiting  changes  only  to  nec- 
essary machine- dependent  aspects.  Changes  are  also  required 
to  add  new  capabilities,  correct  errors  in  the  program,  and 
improve  software  performance.  Different  approaches  are  nec- 
essary to  satisfy  these  different  types  of  modif iaoility 
f 26  1. 


SYSTEM 
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Figure  5.  Software  Reliability  Oriented  Life  Cycle  Plan 
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Additionally,  aodif iability  implies  not  only  the 
ability  to  have  an  adaptaole  evolutionary  desiqn,  employ 
standardized  software  buildinq-blocks , tune  for  performance, 
etc.,  out  also  the  ability  to  maintain  project  schedules  and 
budqets.  There  has  been  much  proqress  in  achievinq  this 
qoal  within  tne  past  ten  years. 

Sf f iciencv . defined  as  the  optimal  use  of  computer  re- 
sources by  a proqrara,  is  a much  abused  qoal.  Primarily  this 
is  because  it  is  prematurely  assiqned  a hiqh  priority  in  en- 
qineerinq  tradeoffs.  Efficiency  should  ae  treated  within 
the  context  of  other  issues.  For  example,  achievinq  modifi- 
ability can  provide  the  basis  for  meetinq  efficiency  qoais 
durinq  the  maintenance  phase  of  the  software  life  cycle.  In 
addition,  insiqhts  reflectinq  a more  unified  understandinq 
of  a problem  have  more  impact  on  efficiency  (via  aostractioa 
and  uniformity)  than  any  amount  of  "bit  twiddlinq"  within  a 
faulty  structure.  In  qeneral,  the  efficiency  qoal  does  not 
dominate,  as  reliability  and  modifiability,  the  practice  of 
software  enqineerinq  f26]. 

Reliabilit y is  an  important  qoal  which  is  much  in  voque 
today.  Reliability  is  concerned  with  conception,  desiqn, 
and  construction  as  well  as  failure  in  operation  or  perform- 
ance. Unlike  efficiency  which  is  often  prematurely  applied, 
reliability  is  more  often  considered  too  late  in  the  soft- 
ware life  cycle.  Since  reliability  can  only  be  built  in  at 
the  beqinninq  of  the  development  cycle — it  cannot  ae  an  add- 
on at  the  end — it  is  a primary  problem  to  be  solved  in  any 
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software  system.  Hence,  reliability  has  a crucial  effect  on 
software  enqineerinq  practices  [ 26  1.  Because  of  its  impor- 
tance, Chapter  III  will  be  devoted  entirely  to  it. 

Under stan lability  is  the  final  basic  qoal  which  exerts 
a stronq  influence  in  all  aspects  of  software  enqineerinq. 

In  particular,  it  is  not  a property  of  leqaiity.  It  is, 
therefore,  much  more  important  since  the  entire  conceptual 
structure  is  involved  [26,271.  Also,  in  any  circumstance  an 
acceptable  level  of  understandability  either  is  or  is  not 
present.  Thus,  there  is  no  middle  qround.  Althouqh  under- 
standability is  a prerequisite  to  reliability  and  modifi- 
ability, it  also  draws  attention  to  an  important  barrier  to 
it — complexity  [26].  Manaqement  of  complexity  is  a crucial 
part  of  software  enqineerinq  methods,  and  the  need 
to  manaqe  complexity  arises  from  the  qoal  of  understandabil- 
ity. The  only  way  to  achieve  understandability  relative  to 
an  inherently  complex  system  is  to  impose  an  appropriate 
structure  and  orqanizaticn  on  the  software  system.  As  such, 
the  structure  must  be  represented  in  a clear  notion  that 
permits  the  different  translations  (requirements,  desiqn, 
source  codinq,  object  codinq,  and  documentation)  to  oridqe 
the  qap  between  the  actual  system  and  an  understandable  rep- 
resentation of  it.  Thus,  achieviuq  understandaoility 
depends  as  much  upon  the  software  enqineerinq  toois  such  as 
compilers  as  on  the  methods  such  as  structured  proqraaminq 


f 26  1. 
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Other  qoals  such  as  portability  and  testability  are  or 
lesser  importance  than  the  ones  discussed  above.  Boehm 
f 27,281  discusses  all  of  the  above  qoals  as  characteristics 
of  quality  software. 


The  Principles  of  Software  Engineering 

The  principles  of  software  engineering  are  modularity, 
abstraction,  localization,  hiding,  uniformity,  completeness, 
and  confirmability.  These  principles  are  applied  in  various 
combinations  throughout  the  fundamental  software  life  cycle 
(see  Figure  4)  to  achieve  the  desired  goals  discussed  aoove 
r 5,251. 

The  decomposition  of  a system  [29]  depicts  the  programs 
or  modules  of  the  system  organized  into  a structure  by  the 
relationships  (interfaces)  among  them.  The  seven  princi- 
ples, singly  and  in  combination,  are  used  to  determine  and 
control  those  relationships  [26].  They  are  used  as  decision 
criteria  to  ensure  that  the  resulting  decomposition  attains 
the  goals  of  the  software  system.  Thus,  each  principle 
deals  with  some  aspect  of  the  relationships-i. e. , the 
interfaces  among  the  modules  or  programs.  The  rest  of  tnis 
section  discusses  each  principle  separately. 

Modularity  deals  with  the  properties  of  a hierarchical 
software  structure.  It  has  been  given  various  definitions 
by  several  authors  [30-36].  Basically  modularity  deals  with 
how  the  structure  of  an  object  can  make  the  attainment  of 
some  purpose  easier.  In  essence,  modularity  is  purposeful 
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structuring  [ 29  ].  Therefore,  the  principle  of  modularity  is 
made  concrete  by  explaining  how  certain  constraints  on  the 
structure  of  systems  can  make  it  easier  or  harder  to  achieve 
some  goal  such  as  modifiability,  efficiency  or  reliability. 

Imposing  constraints  on  structures  is  the  essence  of 
applying  the  modularity  principle  in  software  engineering 
f 26  ].  For  example,  top-down  structured  programming  [ 3o  ] 
which  forces  programmers  to  make  explicit  the  conditions 
under  whicn  programs  are  designed  and  coded  can  help  ensure 
unierstandability  and  prevent  errors  [26]. 

It  may  be  possible  for  a given  program  to  satisfy  all 
goals  simultaneously.  A program  may  have  one  structure  if 
modules  are  constructed  according  to  cne  rule  (module 
strength)  and  a different  structure  if  a different  rule 
(module  coupling)  is  considered  [4,37]. 

Abstraction  is  a very  pervasive  principle  [34,21]. 
Despite  the  existence  of  the  above  papers,  no  practical  def- 
inition of  abstraction  exists.  However,  most  researchers  in 

<1 

this  field  agree  that  the  essence  of  abstraction  is  to  ex- 
tract essential  properties  while  omitting  nonessential 
details.  Hierarchical  decomposition  in  the  form  of  levels 
shows  abstraction  in  its  best  form.  Each  level  or  tne 
decomposition  shows  an  abstract  view  of  the  lower  levels 
purely  in  the  sense  that  details  are  subordinated  to  the 

lower  levels  f 2 6 1 . The  top  level  expresses  the  program  in 
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teens  natural  to  the  oriqinator  of  the  task  while  lower 
levels  express  commitment s to  specific  ways  of  realizing  tne 
terms  of  the  higher  levels  f 39  ]. 


When  combined  with  the  principle  of  completeness,  ab- 
straction ensures  that  a given  level  in  a decomposition  is 
understandable  as  a unit  without  requiring  either  knowledge 
of  the  lower  levels  of  detail,  or  necessarily  how  it  partic- 
ipates in  the  system  as  viewed  from  a higher  level.  As 
such,  this  principle  is  employed  on  the  one  hand  to  obtain  a 
description  of  some  level  of  the  system  which  could  be  real- 
ized by  any  of  several  implementations,  and  on  the  other 
hand  to  give  a description  of  one  part  of  a system  which 
could  be  used  in  many  other  systems  requiring  the  same  com- 
ponent at  that  level  of  abstraction. 

Abstraction  interacts  strongly  with  the  purpose 
underlying  any  particular  decomposition.  Unless  it  is  com- 
bined with  the  principle  of  modularity,  abstraction  is  of 
little  practical  value.  When  employed  to  achieve  the  goal 
of  understandability,  each  decomposition  level  while 
presenting  more  and  more  detailed  views  of  the  system  must 
do  so  in  terms  that  are  understandable  to  the  intended  user 

r 26 1. 

Local izat ion  is  concerned  with  physical  proximity. 
Things  must  be  brought  together  in  one  place.  Thus,  the 
localization  principle  deals  with  physical  interfaces, 
textual  sequence,  memory,  etc.  The  other  principles  can 
interrelate  the  localized  things  to  serve  specific  purposes. 
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Loqical  aad  physical  records  as  hell  as  paged  memories 
are  examples  of  localization.  Also  the  avoidance  of  GGTO's 
in  structured  programming  is  an  application  of  localization 
to  control  structures  which  simplifies  confirmability  aad 
enhances  understandaoilit y [26]. 

The  Hiding  principle,  as  discussed  oy  parnas  f29],  is 
used  as  the  major  criterion  for  a decomposition  into 
modules.  Althouqn  it  is  not  the  same,  it  is  related  to  the 
idea  of  postponing  bindinq  decisions  in  top-down  program- 
ming. The  purpose  of  hidinq  is  to  make  visible  only  those 
properties  of  a module  needed  to  interface  with  other 
modules  and  to  make  inaccessable  details  that  should  not 
affect  other  parts  of  a system.  Abstraction  assists  in 
identifying  details  that  should  be  hidden.  Basically, 
hiding  is  concerned  with  access  constraints  i.  29  ]. 

Uniformity  is  also  an  important  principle.  Since  it 
ensures  consistency,  it  is  an  obvious  principle  to  apply  in 
software  enqineerinq.  It  is  applied  to  notational  matters 
to  yield  notation  (documentation)  that  is  free  of  confusing 
and  perhaps  costly  inconsistencies.  When  c cnbined  with  the 
abstraction  principle,  uniformity  implies  a notation  tnat 
permits  arbitrary  mechanization  of  the  internal  detailing  of 
an  object  (the  notation  does  not  constrain  one's  choice  of 
implementation) . Also,  when  the  hidinq  principle  is  added, 
the  result  is  a notation  that  does  not  permit  several  imple- 
mentation choices  and  also  ensures  that  no  unnecessary 
details  of  specific  implementaion  are  revealed  by  the  nota- 
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tion.  Basically,  uniformity  is  the  lack  of  inconsistencies 
and  unnecessary  differences  [26]. 

Completeness  is  another  obviously  important  principle. 
This  principle  ensures  that  all  the  essentials  of  an  ao- 
straction  are  explicit  and  that  nothinq  essential  is  left 
out.  Every  detail  does  not  have  to  be  shown,  but  tne  set  of 
abstract  concepts  must  cover  every  detail. 

When  completeness  is  applied  to  notational  matters,  it 
requires  that  a notation  provides  a means  for  sayinq 
everythinq  that  one  wants  to  say.  When  it  is  combined  with 
abstraction,  completeness  implies  that  a notation  should  be 
concise,  permittinq  the  suppression  of  invariant  details  in 
favor  of  hiqhliqhtinq  the  chanqeable  details.  Additionally, 
completeness,  when  combined  with  uniformity  and  abstraction 
and  applied  to  the  qoal  of  efficiency,  allows  proqrammers  to 
select  different  implementation  mechanisms  to  tune  a sys- 
tem's performance  without  havinq  to  chanqe  the  form  of  any 
subroutine  call  r 261. 

Confirmability  is  a principle  that  ensures  that  infor- 
mation needed  to  verify  correctness  has  been  explicitly 
stated.  This  information  is  used  for  finainq  out  whether 
stated  qoals  such  as  reliability  have  been  achieved. 

"Applied  to  desiqn  issues,  confirmability  re- 
fers to  the  structurinq  of  a system  so  it  is 
readily  tested.  It  must  be  possible  to  stim- 
ulate the  constructed  system  in  a controlled 
manner  so  its  response  can  be  evaluated  for 
correctness.  Applied  to  notational  matters, 
confirmability  means  that  a notation  should 
require  explicit  specification  of  constraints 
that  affect  the  correctness  of  a desiqn  or  im- 
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plementation  (e. g. , data  declarations  that 
specify  ranqe  of  values  and  units  of  value 
as  well  as  mode  of  representation) . Applied 
to  the  practice  of  software  engineering,  con- 
firmability refers  to  the  use  of  such  metnods 
as  structured  walk-throughs  of  design,  egoless 
proqramminq  T331,  and  other  methods  that  help 
to  ensure  that  nothing  has  been  overlooked."  f 26  ]. 

Software  ae t r ics 

The  result  of  effective  software  engineering  is  the 
production  of  a program  that  meets  the  requirements 
(assuming  the  requirements  are  accurately  stated)  of  the 
user.  But , how  can  software  be  measured  so  it  can  be 
compaired  aqainst  specified  goals  of  the  user?  Currently, 
measures  of  software  attributes  seem  to  be  an  answer. 

The  term  "metric"  by  definition  means  a standard  of 
measure.  A software  metric  is  defined  as  a measure  of  the 
extent  or  deqree  to  which  software  possesses  and  exhibits  a 
certain  property  or  attribute  [27,28,40].  Software  metrics 
is  discussed  briefly  in  the  following  paragraphs.  Chapter 
IV  will  concentrate  on  the  metrics  applied  to  C030L  source 
code  as  a measure  of  proqram  composition. 

It  seems  obvious  that  the  software  profession  is  at  the 
point  of  moving  from  a handicraft  into  an  engineering  indus- 
try. There  have  been  enough  larqe  failures  in  software  pro- 
jects to  motivate  us  to  acquire  full  control  over  the  soft- 
ware technology.  To  be  successful  and  have  full  control,  we 
must  be  able  to  recoqnize  and  measure  all  critical  factors, 
and  not  simply  the  easily  available  ones,  such  as  space  and 
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time  consumption.  Software  metrics  is  concerned  with  aeas- 
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urinq  all  factors,  simple  and  critical,  related  to  software. 
In  particular,  measures  relating  to  the  use  of  human  talent 
resources  are  of  magor  interest  because  of  its  scarcity 
today,  compared  to  the  relatively  cheap  machine  resources. 
Also,  measures  related  to  reliability  are  becoming  more  and 
more  important  as  computers  are  increasingly  used  for  cru- 
cial functions  f 4 0 1.  There  are  many  other  software  metrics 
(see  Fiqure  6)  such  as  maintainability,  portability,  under- 
standability , etc.,  but  this  paper  is  concerned  with  measur- 
ing one  characteristic — internal  complexity  of  COBOL  pro- 
grams. These  metrics  will  be  discussed  in  detail  in  chapter 


Summary 

There  are  many  aspects  of  software  engineering.  The 
intent  of  this  chapter  has  been  to  focus  the  underlying 
goals  and  principles  of  software  engineering  into  a coherent 
framework  for  the  readers  of  this  paper.  Software  metrics 
is  applied  to  determine  to  what  degree  a certain  attribute 
is  present  in  software. 
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What  is  Software  Reliability? 

The  most  siqaificant  problem  facing  the  software  pro- 
fessional today  is  unreliable  software.  This  is  the  reason 
for  recent  emphasis  on  developing  a software  reliaoility 
theory.  As  previously  defined,  software  reliability  is  tne 
probability  that  a given  program  operates  correctly,  without 
an  error,  for  some  time  period  on  the  machine  for  which  it 
was  designed.  Software  reliability  is  thus  a function  of 
the  number  of  errors  in  a program. 

Reliability  is  not  an  inherent  property  of  a program; 
it  is  largely  related  to  how  the  program  was  designed,  con- 
structed, tested,  and  operated.  The  word  probaoility  in 
the  definition  actually  represents  the  probability  that 
there  are  no  errors  in  the  proqram  qiven  a valid  input  from 
its  input  space.  At  times  it  is  simply  used  as  a qualita- 
tive measure  of  the  lack  of  errors  in  a proqram  [4], 

Reliability  as  a Measure  of  Software  Quality 
To  provide  a meaninqful  assessment  of  software  quality, 
quantitative  methods  of  evaluating  software  are  being  devel- 
oped. Until  recently,  quality  assessments  have  been  subjec- 
tive evaluations  of  software  based  cn  proqram  deficiencies. 
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However,  sub-jective  evaluations  for  software  are  not  con- 
sistent with  the  use  of  the  metnodoloqies  used  to  measure 
the  quality  of  hardware.  For  complex  computer  systems, 
consistinq  of  hardware,  software,  and  human  interface  sua- 
systems,  the  most  meaninqful  aeasure  of  quality  is  total 
system  reliability.  As  such,  the  most  meaninqful  measure  of 
software  quality  is  the  reliability  of  the  software  subsys- 
tem. If  software  reliability  is  not  explicitly  stated  it 
must  be  determined  from  the  specification  of  the  total  sys- 
tem. A study  of  the  total  system  reliability  and  cost- 
benefit  trade-offs  will  determine  the  reliability  apportion- 
ment amonq  the  hardware,  software  and  human  operated  subsys- 
tems f 4 1 1. 

What  is  an  Error? 

Althouqh  software  reliability  is  the  most  appropriate 
aeasure  of  software  quality,  there  are  terminology  problems 
because  the  meaninqs  of  such  words  as  software  failure  and 
software  errors  are  not  entirely  obvious  by  analogy  with  the 
corresponding  hardware  reliability  concepts  which  are  well 
defined.  A software  error  is  present  when  an  input  is  made 
or  a command  is  qiven  and  the  proqram  does  not  respond  as 
the  user  expects  it  to.  A failure  is  an  occurrence  of  an 
error.  A failure  may  be  manifested  in  many  ways.  A com- 
plete stoppaqe  of  the  proqram  may  or  may  not  occur. 

Detection  of  failures  is,  to  a large  extent,  a subjec- 
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tive  decision  which  must  be  made  by  the  users  or  the  test 
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personnel.  Hopefully,  this  decision  will  be  made  on  the 
basis  of  ob-jective  criteria  such  as  performance  specifica- 
tions. In  actual  practice,  failure  detection  depends  on  a 
user’s  observation  of  an  error,  so,  in  effect,  a software 
failure  is  what  a user  says  is  an  error. 

After  failures  are  detected  a programmer  must  analyze 
the  program  and  locate  the  causes  of  the  failure.  Basically 
all  errors  are  design  or  implementation  errors.  Logical  or 
clerical  errors  in  coding  may  be  found  to  be  responsible  for 
producing  the  incorrect  results.  Also  the  program  specifi- 
cation could  be  in  error.  When  errors  are  located,  action 
is  taken  to  correct  the  errors  to  prevent  recurrence  of  tne 
failures.  The  correspondence  between  software  errors  un- 
covered and  software  failures  detected  is  not  necessarily 
one-to-one.  Many  errors  may  occur  without  a failure  being 
detected,  and  a failure  may  be  a result  of  several  errors. 
Also,  a software  failure  may  be  reported  that  is  in  fact  no 
software  failure  at  all,  but  rather  a user  or  hardware  defi- 
ciency. 

Failures  differ  with  respect  to  their  impact  on  tne 
mission  of  the  software.  Severe  failures  may  result  in  a 
failure  of  a mission,  while  less  severe  failures  may  only 
cause  aggravations  or  limitations  which  have  little  effect 
on  the  overall  mission  of  the  total  system. 


The  reader,  if  he  has  written  a large  program,  should 
now  be  able  to  grasp  the  elusive  nature  of  software  reli- 
ability. software  errors  are  not  an  inherent  property  of 
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software.  Errors  are  basically  human  mistakes  and  we  can 
never  expect  to  find  them  all — regardless  of  how  well  we 
test  the  proqrams.  But,  we  can  measure  or  predict  the  num- 
ber of  residual  errors  so  we  can  decide  when  tne  software 
has  reached  an  acceptable  reliability  level. 


Do  Software  Failures  Occur  Randomly  with  Time? 

Unlike  hardware,  there  is  no  physical  mechanism  which 
generates  software  failures.  When  all  errors  are  removed, 
the  software  is  100  per  cent  reliable  and  will  remain  so 
forever,  provided  no  program  changes  are  made.  What  then 
accounts  for  the  randomness  of  software  failures? 

Different  input  combinations  result  in  a different  re- 
sponse from  the  software.  The  paths  traversed  within  a 
software  program  depend  on  the  input  combinations.  Each 
path  can  be  thought  of  as  containing  possible  software  er- 
rors waiting  to  be  discovered.  Without  correction,  the  same 
errors  will  occur  each  time  the  same  logic  path  is  executed. 
If  the  errors  result  in  an  observable  software  failure, 
the  given  failure  can  be  reproduced  at  will,  or  it  can  be 
avoided  by  user  control  of  the  input  combinations.  There- 
fore, software  failures  are  functions  of  the  input 
combinations — not  random  functions  of  time.  However,  in 
reality,  input  combinations  are  chosen  in  a scmewhht  random 
fashion,  and  the  resultant  effect  is  that  errors  are  un- 
covered and  failures  are  observed  at  random.  It  is  with 
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this  meaninq  that  we  talk  anout  the  random  occurrence  of 
software  failures  f 41  1. 

Reliability  Models 

The  most  important  unknown  of  software  reliability  is 
the  number  of  residual  errors  in  a proqram.  If  an  estimate 
of  this  number  were  available  durinq  the  testinq  staqes  it 
would  help  determine  when  to  stop  testinq.  Also  if  we  knew 
the  number  of  reaaininq  errors  in  an  operational  proqram  we 
could  estimate  the  cost  of  maintenance  and  estaolish  a level 
of  confidence  in  the  proqram.  Other  related  attributes  for 
which  estimates  are  desirable  are  the  reliability  of  the 
proqram  and  the  mean- time-to-f ailure  of  the  proqram.  Meas- 
ures of  the  proqram's  complexity  would  be  useful  to  estimate 
the  number  of  errors  and  to  judge  the  quality  of  the  desiqn. 
If  software  reliability  models  were  available  that  would 
model  software  failures,  then  one  could  deal  with  the 
unknowns  of  software  reliability  [4]. 

There  are  3 types  of  software  reliability  models  being 
evolved  today.  A number  of  software  reliability  models  are 
discussed  in  references  [ 42-47  1.  Tnese  models  are  closely 
related  to  hardware  reliability  theory  and  contain  signifi- 
cant assumptions  about  the  underlying  probability  distribu- 
tion of  software  failures.  References  [48-56]  are  reli- 
ability studies  which  contain  evaluations  of  these  models 
relative  to  specific  error  data.  These  models  seem  to  apply 
only  to  specific  situations  and  do  not  have  general  applica- 
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tion  in  any  environment.  The  next  set  of  models,  discussed 
in  references  f 11,57-591,  produce  similar  results,  but  are 
not  based  on  hardware  reliability  theory.  The  last  set  of 
models  is  concerned  with  predicting  the  complexity  of  a pro- 
gram r 14-22  1.  Tne  rest  of  this  chapter  will  discuss  tne 
hazard  function  for  software  failures. 

Hazard  Function  for  Software  Failures 

Me  shall  assume  that  a large  proqram  is  resident  iu  a 
computer  and  is  servicing  a steady  stream  of  dissimilar  "in- 
puts". Me  shall  assume  that  these  inputs  enter  the  proqram 
at  arbitrary  points  in  time,  and  that  each  such  entry  can  be 
looked  upon  as  an  opportunity  to  detect  an  error  in  tne  pro- 
qram. Thus,  we  assume  that  software  errors  are  detected  in 
a random  manner. 

Software  dees  not  aqe  with  time,  therefore,  it  is  rea- 
sonable to  assume  that  its  failure  rate  is  constant  between 
points  in  time  at  which  changes  are  made.  Every  time  an  er- 
ror is  detected,  we  eliminate  it.  If  we  ignore  tne  possi- 
bility of  introducing  new  errors  then  our  failure  cate  is  a 
step  function  as  indicated  in  Figure  7.  Several  variations 
and  -justifications  for  this  model  have  appeared  in  tne  lit- 
erature. For  the  purpose  of  this  paper,  it  will  suffice  to 
illustrate  that  a proqram  failure  rate  is  decreasing  and 
will  eventually  qo  to  zero,  assuming  there  are  no  moaitLCa 
tions  for  new  capabilities.  This  is  in  contrast  to  the 
hazard  function  of  a hardware  component  (see  Figure  d) . 


3 


I 


\ 


35 


At  the  start  of  the  testinq  process,  we  assume  that  the 
proqram  contains  an  unknown  number  of  errors,  say  N.  The 
failure  rate  is  assumed  to  be  proportional  to  the  residual 
number  of  errors  in  the  proqram.  Every  tine  an  error  is  en- 
countered, the  error  is  removed  and  nc  new  error  is  intro- 
duced- Althouqh  these  assumptions  make  the  model  less  than 
realistic,  Jelinski  and  Moranda  £ 42  ] have  demonstrated  the 
usefulness  of  a model  of  this  type  in  analyzing  error  data 
from  the  U.S.  Navy  and  the  Apollo  proqram  of  NASA- 

Let  x(1)  , x (2) , ...  , denote  the  points  in  time  at 
which  software  errors  are  detected  and  corrected.  Then,  ac- 
cordinq  to  the  assumptions  of  the  model,  the  failure  rate 
between  x(i-1)  and  x(i)  is  K(N-i  + 1),  i = 1,2,  ...  , n,  for  n 
< N,  where  K is  the  constant  of  proportionality.  The  fail- 
ure rate  generated  by  the  testinq  process  is  illustrated  in 
Fiqure  7.  If  T { i)  denotes  the  time  interval  between  x ( i—  1 ) 
and  x(i),  then  from  the  assumption  that  time- to-f ail ure  is 
exponentially  distributed 


?rt(i)  1 = Pf  T (i)  <t  (i)  1=1-  expf-K  (N-i+1 ) t (i)  J,  K > 0, 
and  t(i)  > 0. 

If  K and  N were  known,  then  the  reliability  of  the 
software  prior  to  testing,  the  distribution  function  of  the 
time  to  test,  the  number  of  errors  to  be  removed  in  order  to 
obtain  a desired  level  of  reliability,  and  other  such  infor- 
mation can  easily  be  determined.  In  practice,  K and  N are 
unknown,  and  hence  the  estimation  of  K and  N becomes  criti- 
cal. 


....  . 
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In  order'  to  estimate  K and  N,  and  to  obtain  a stopping 
rule  for  testinq  the  program,  cne  must  use  t(1),  t(2),  ...  , 
t{n)  as  the  realizations  of  T(1),  T(2),  ...  , T(n),  for  n < 
N.  This  estimation  problem  is  an  unusual  cne.  The  time  in- 
tervals t(1),  t (2) , ...  , t (n) , do  not  constitute  a random 
sample  of  size  n from  a single  failure  distribution,  out 
rather  n samples,  each  of  size  1,  from  n different  but  re- 
lated distributions  r 101.  Therein  lies  the  problea  of  pre- 
dicting K and  N. 

Schneidevind  f 60, 611  shoved  that  error  data  fitted  no 
single  underlying  probability  distribution.  This  is  more 
reason  to  believe  that  failure  functions  are  not  only  func- 
tions of  the  remaining  errors  but  also  the  composition  of 
the  program  itself. 

The  remaining  chapters  of  this  paper  will  be  devoted  to 
developing  and  discussinq  a model  tor  estimating  the  number 
of  errors  in  COBOL  programs. 


IV.  DESCRIPTION  OF  DATA 


Introduction 

Error  data  processinq  involves  three  interrelated  ac- 
tivities: collection,  classification  into  error  categories, 
and  analysis.  Since  collection  and  classification  have  Deen 
dealt  with  adequately  in  ether  sources  r 61  — 67  1 [50]  [561, 
analysis  is  the  primary  concern  of  this  paper.  While  analy- 
sis has  been  the  primary  concern  of  this  research,  the  other 
two  were  considered  by  usinq  concepts  and  techniques  devel- 
oped in  other  studies. 

The  purpose  of  this  chapter  is  to  describe  data  that 
are  used  to  develop  a model  for  predicting  the  number  of  er- 
rors in  COBOL  proqrams.  Chapter  V will  present  a detailed 
analysis  of  the  data  described  in  this  chapter. 

Terminology  Revisited 

Althouqh  defined  elsewhere,  several  terms  need 
clarification  to  familarize  the  reader  with  what  follows  in 
the  rest  of  this  paper. 

The  term  software  reliability,  for  the  purpose  of  the 
analysis  of  empirical  data,  needs  to  be  redefined.  Software 
posseses  reliability  to  the  extent  that  it  is  expected  to 
perform  its  intended  functions  satisfactorily.  With  tnis  in 
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mind,  errors  in  proqrams  represent  an  inability  of  the  pro- 
qrams  to  perforin  intended  functions  satisfactorily.  An 
error-free  proqram  would  be  a reliable  proqram.  Henceforth 
in  this  paper,  anythinq  that  causes  software  not  to  perform 
its  intended  function  is  an  error.  Specifically,  the  term 
error  is  a user  dissatisfaction,  which  is  documented  on  a 
fora,  with  the  results  of  a proqram.  The  error  may  not  nec- 
essarily be  the  result  of  an  execution  of  a proqram,  e. q. , 
desiqn  reviews  can  result  in  the  detection  of  errors. 

The  term  project  is  the  combination  of  development  ac- 
tivities required  to  produce  the  software  and  its  documenta- 
tion. Three  sources  of  data  are  used  in  this  report.  Be- 
cause of  the  restrictions  in  employinq  the  actual  system  and 
proqram  names,  the  data  sources  are  called  Project  1,  Pro- 
ject 2,  and  Project  3.  Each  one  will  be  discussed  later. 


Project  Descriptions 

The  three  projects  represent  small  to  larqe  software 
development  activities.  The  application  software  for  all 
three  projects  is  written  in  COBOL.  The  smallest  compilable 
unit  of  source  code  is  the  program.  Each  project  is  dis- 
cussed below.  Table  1 lists  the  data  available  from  each 
project. 


££2je£t_J 

Project  1 is  a data  collection  system  [67]  consistinq 
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of  5 batch  proqrams  with  a total  of  2280  lines  of  coda. 

The  system  provides  an  on-qoinq  data  tase  for  input  into  re- 
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TABLE  1.  Data  Availability  for  Each  Project 


Project 

1 

Project 

2 

Project 

3 

1) 

General  Project 
Descriptions 

X 

X 

X 

2) 

Design  Problem  Data 

X 

X 

3) 

Problem  Report 
(Error)  Data 

X 

X 

X 

4) 

Software 

Characteristics 

X 

X 

X 

5) 

Testing  Data 

X 

'!  X 

6) 

Computer  Usage  Data 

X 

X 
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liability  models.  The  data  base  also  contains  program  cnar- 
acteristics  as  discussed  in  this  paper.  The  system  applies 
to  C030L  proqrams  designed  to  execute  on  the  Honeywell  H6Q60 
computer  system  throughout  the  Air  Force.  The  5 proqrams  in 
this  system  utilize  a file  management  system  availaole  on 
the  H6060. 


££2ject_2 

Project  2 is  an  on-line  system  involving  several  kinds 
of  data  processing  activities  such  as  personnel  management, 
accounting  and  finance,  inventory  etc.  Only  14  programs  are 


available  for  analysis.  There  are  19045  lines  of  source 
code  in  these  programs.  These  proqrams  execute  on  the 
National  Cash  Register  NCH8200  computer  system. 


££Oject_j 

Project  3 represents  an  initial  delivery  of  a large  on- 
line Command  Manpower  Data  S ystem  (CMDS) . CMOS  is  a resource 
accounting  and  management  information  system  which  supports 
the  Manpower  and  Organization  function  at  Major  Command 
level  throughout  the  Air  Force.  Data  for  46  proqrams  ace 
available  for  analysis.  There  are  54116  lines  of  source 
code  in  these  programs.  These  programs  execute  on  the  HnObO 


computer  system  and  perform  a wide  variety  of  data  process- 
ing activities,  general  purpose  utility,  data  retrieval. 


data  maintenance,  etc. 
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Approach  to  Data  collection  aad  Classification 
It  would  be  ideal  to  perform  a study  of  this  nature 
usinq  the  same  collection  and  classification  tools  and  pro- 
cedures for  all  projects.  Since  real  data  from  on-going 
projects  within  different  organizations  are  being  used,  this 
was  not  possible.  Data  sources  are  the  normal  data 
collection  and  classification  system  of  the  organization 
developing  the  software.  For  example,  the  Air  Force  Data 
System  Design  Center  (AFDSCC1  has  a manual  system  for 
collecting  error  reports.  Project  3 data  was  recorded  using 
this  system. 

Although  the  data  is  reasonably  good,  it  is  onvious 
that  it  is  not  the  same  type  of  data  from  all  projects. 

This  presented  a problem  when  trying  to  classify  an  error 
according  to  a specific  category.  Finally  it  was  decided  to 
work  only  with  actual  errors  that  required  a change  in 
source  code  to  affect  corrective  action.  By  considering 
only  code  change  errors  and  performing  analysis  at  the  indi- 
vidual proqram  level,  it  was  possible  to  generate  similiar 
data  from  all  projects.  Errors  were  classified  into  12  cat- 
egories. These  categories  are: 

1)  Coaputationa, 

2)  Logic, 

3)  Data  Input, 

4)  Data  Handling, 

5)  Data  Output, 

6)  Interface, 
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7)  Array  Processinq, 

8)  Data  Base, 

9)  Operation, 

10)  Proqram  Execution, 

11)  Documentation,  and 

12)  Other 

The  cateqories  alonq  with  types  of  errors  in  each  cate- 
qory  are  presented  in  Table  2. 

Software  Characteristics 

Boehm  T 27  1 presents  a detailed  discussion  of  character- 
istics of  software  qualify  (see  Fiqure  6).  Thayer  and  Lipow 
T 5 0 1 discuss  the  two  forms  of  software  quality  characteris- 
tics, those  that  can  be  quantitatively  measured  and  those 
that  require  some  subiective  evaluation.  Both  are  needed  to 
explain  errors.  Both  forms  were  considered  by  Thayer  and 
Lipow  and  examples  are  presented  in  Tade  3.  The  subjective 
form  did  not  show  much  promise  as  predictor  variables  for 
the  number  of  errors  in  proqrams.  Since  previous  research 
showed  that  software  structure  influenced  the  numoer  of  er- 
rors and  since  our  primary  objective  is  to  develop  a com- 
plexity model  for  predicting  the  number  of  errors  in  COBOL 
proqrams,  this  paper  is  concerned  with  only  structural  char- 
acteristics. Only  those  that  can  be  measured  are  considered 
in  this  report. 
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TABLE  2.  Error  Categories 


COMPUTATIONAL  ERRORS 

Incorrect  operand  in  equation 
Incorrect  use  of  parenthesis 
Sign  convention  error 
Units  or  data  conversion  error 
Computation  produces  an  over/under  flow 
Incorrect/inaccurate  equation  used 
Precision  loss  due  to  mixed  mode 
Missing  computation 
Rounding  or  truncation  error 

LOGIC  ERRORS 

Incorrect  operand  in  logical  expression 
Logic  activities  out  of  sequence 
Wrong  variable  being  checked 
Missing  logic  or  condition  tests 
Too  many/ few  statements  in  loop 
Loop  iterated  incorrect  number  of  times 
(including  endless  loop) 

Duplicate  logic 

DATA  INPUT  ERRORS 

Invalid  input  read  from  correct  data  file 
Input  read  from  incorrect  data  file 
Incorrect  input  format 
Incorrect  format  statement  referenced 
End  of  file  encountered  prematurely 
End  of  file  missing 

DATA  HANDLING  ERRORS 

Data  file  not  rewound  before  reading 

Data  initialization  not  done 

Data  initialization  done  improperly 

Variable  used  as  a flag  or  index  not  set  properly 

Variable  referred  to  by  the  wrong  name 

Bit  manipulation  done  incorrectly 

Incorrect  variable  type 

Data  packing/unpacking  error 

Sort  error 
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TABLE  2.  Error  Categories  (Continued) 


DATA  OUTPUT  ERRORS 

Data  written  on  wrong  file 

Data  written  according  to  the  wrong  format  statement 

Data  written  in  wrong  format 

Data  written  with  wrong  carriage  control 

Incomplete  or  missing  output 

Output  field  size  too  small 

Line  count  or  page  eject  problem 

Output  garbled  or  misleading 

INTERFACE  ERRORS 

Wrong  subroutine  called 

Call  to  subroutine  not  made  or  made  in  wrong  place 
Subroutine  arguments  not  consistent  in  type,  units, 
order,  etc. 

Subroutine  called  is  nonexistent 
Software/data  base  interface  error 
Software  user  interface  error 
Software/software  interface  error 

ARRAY  PROCESSING  ERRORS 

Data  not  properly  defined/dimensioned 
Data  referenced  out  of  bounds 
Data  being  referenced  at  incorrect  location 
Data  pointers  not  incremented  properly 

DATA  BASE  ERRORS 

Data  not  initialized  in  data  base 
Data  initialized  to  incorrect  value 
Data  units  are  incorrect 

OPERATION  ERRORS 

Operating  system  error  (vendor  supplied) 

Hardware  error 

Operator  error 

Test  execution  error 

User  misunderstanding/error 

Configuration  control  error 

PROGRAM  EXECUTION  ERROR 

Bad  object  code 
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TABLE  2 . Error  Categories  (Continued) 


DOCUMENTATION  ERRORS 
User  manual 

Interface  specification 
Design  specification 
Requirements  specification 
Test  documentation 

OTHER 

Time  limit  exceeded 
Core  storage  limit  exceeded 
Output  line  limit  exceeded 
Compilation  error 

Code  or  design  inefficient/not  necessary 

User/programmer  requested  enhancement 

Design  nonresponsive  to  requirements 

Code  delivery  or  redelivery 

Software  not  compatible  with  project  standing 


TABLE  3.  Available  Parameters 


Program  Structural  Characteristics 
Program  size 

Total  source  code  statements 
Executable  statements 
Non-executable  statements 
Machine  dependent  number  of  instructions 
(ENTER  SYMBOLIC) 

Number  of  unconditional  branches 

Number  of  conditions  in  program 

Number  of  direct  interfaces 

With  routines  within  program  and  other  applicati 
programs 

With  operating  system 

Number  of  arguments  in  interface  calls 

Data  interfaces 

Number  of  global  data  blocks 
Number  of  internal  data  variables 

Number  of  procedures 

Number  of  entry  points 

Number  of  exit  points 

Routine  code  type 

Number  of  computational 
Number  of  logical 
Number  of  data  handling 
Number  of  I/O 

Loop  and  nesting  levels 

Pages  of  documentation 
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TABLE  3.  Available  Parameters  (Continued) 


Computer  time  (clock  time,  not^  CPU  time) 

Development  time 
Test  time 


Subjective  Characteristics 


Routine  difficulty  at  preliminary  design 

Routine  difficulty  after  formal  test  and  delivery 

Design 

Code 

Debug/checkout 

Implementation 

Documentation 

Routine  type 

Executive 

Control 

Setup 

Input 

Computational 
Post  processing 
Output 

Personnel  data 

Number  of  people  working  on  routine 
Load  factor  on  each  programmer 
Programmer  rating 
Programmer/ job  evaluation 


Structural  Characteristic s 

Proqram  structural  characteristics  are  measurable. 

They  quantify  the  actual  physical  attributes  of  a proqram. 
The  application  of  metrics  allows  the  quantif ication  or  such 
thinqs  as  a proqram' s size,  input/output  pa t terns,  use  of  a 
data  base,  computations  performed,  interfaces,  use  or  the 
various  lanquaqe  elements,  and  loqical  complexity 

The  approach  taken  was  to  provide  as  mucn  quantative 
detail  as  possible.  In  an  effort  to  tie  specific  error  cat- 
eqories  to  types  of  code  within  a proqram,  22  qeneric  types 
of  structural  characteristics  were  chosen  as  lanquaqe  met- 
rics. The  structural  characteristics  chosen  for  this  study 
are  presented  in  Table  4.  Please  note  that  a measure  for 
each  error  cateqory  is  included.  The  purpose  tor  cnoosinq 
these  characteristics  is  to  measure  the  likelihood  that  a 
proqram  may  have  particular  kinds  of  errors.  These 
characterisitcs  will  also  be  useful  in  future  studies  of  er- 
ror type  distributions.  Since  there  were  no  automated  tools 
available  to  collect  structure  data,  a proqram  was  developed 
by  the  author  to  analyze  COBOL  source  code.  This  proqram  is 
called  COBOL  Characteristics  Analyzer  Proqram  (CCA) . It  was 
ociqinally  desiqned  for  the  NCR3200  computer  system,  and  has 


been  converted  to  run  on  the  H6060  computer 


TABLE  4.  STB  UCTU  RAL  CHAU  ACT  ERISTICS  DEFINITIONS 


Metric 

Variable 


EXIT 


STOP 


NNEX 


Definitions 


Number  of  loqical  conditions 

Number  of  input/output  statements 

Number  of  arithmetic  statements 

Number  of  data  transfer  statements 

Number  of  CALLS  to  external  and 
internal  routines 

Number  of  unconditional  branches 

Number  of  EXIT  statements 

Number  of  STOP  statements 

Number  of  CALLS  to  operating  system 

Number  of  CALLS  to  compiler  to  COPY 
source  code  from  the  library 

Total  statements  = NEX  + NNEX 

Number  of  executable  statements 

Number  of  non-executable  statements 

Number  of  file  descriptions 

Number  of  record  descriptions  or  ”01"  level 
descriptions 

Number  of  data  item  descriptions 
Total  descriptions  = FD  ♦ RD  ♦ DD 
Number  of  data  references 
Number  of  comments 
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TABLE  4-  STRUCTURAL  CHARACTERISTICS  DEFINITIONS 

cont inued 


Metric 

Variable 


Definitions 


Number  of  paragraphs 


Number  of  source  lines 

There  can  be  more  than  one  statement  per  line. 
Number  of  references  to  "reserved"  words 


COBOL  Characteristics  Analyzer  Program 

CCA  is  a utility  proqram  which  statistically  analyzes 
COBOL  source.  It  breads  a proqram's  code  into  its  language 
elements.  This  analysis  is  done  at  the  proqram  level;  how- 
ever, it  identifies  interfaces  between  routines,  between  the 
subject  proqram  and  other  application  programs,  and  between 
the  subject  proqram  and  the  operating  system.  Table  4 nas 
presented  the  list  of  metrics  chosen  to  quantify  the  struc- 
tural characteristics  of  COBOL  proqrams.  CCA  computes  the 
values  for  these  metrics.  Figure  9 presents  sample  output 
for  a proqram  called  S-PTUO. 

Please  note  that  the  columns  PERCENT  OF  TOTAL  and 
PERCENT  OF  EXECUTABLE  (see  Figure  9)  require  special  inter- 
pretation. For  example,  the  number  of  logical  is  30.  This 


is  not  the  number  of  logical  statements  in  the  proqram,  it 


represents  the  number  of 


:ond itions  in  the  proqram. 


Each  AND  and  OR  is  counted  as  a ioqicai  condition.  Because 
of  this  the  PERCENT  columns  will  not  add  to  100  percent. 


Summary  of  Available  Data  for  Each  Project 
Individual  project  data  is  summarized  in  Taoles  5-7  re- 
spectively. Tables  8-10  contain  descriptive  statistics  for 
individual  project  data.  The  error  data  were  collected  from 
software  discrepancy  reports  provided  by  project  program- 
mers. Program  CCA  was  used  to  collect  the  structural  char- 
acteristics data.  This  data  is  analyzed  in  the  next 
chapter.  When,  in  the  course  of  analysis,  specific  project 
data  are  germane  to  results,  the  reader  is  encouraged  to 
refer  back  to  the  corresponding  data. 
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V.  ERROR  DATA  ANALYSIS 

Introduction 


Error  data  analysis  is  important  because  of  the 
necessity  to  cope  with  problems  of  cost  and  software  unreli- 
ability. Examples  of  areas  which  benefit  from  error  data 
analysis  include  management  of  software  development  efforts, 
desiqn  of  software  engineering  technigues  and  tools,  and 
determining  which  software  characteristics  are  relevant  to 
software  reliability.  This  chapter  is  mainly  concerned  witn 
the  latter.  The  principal  emphasis  of  this  analysis  has 
been  on  individual  program  error  data  collected  during 
testing  and  operational  usaqe. 

There  are  many  kinds  of  data  available  from  the  soft- 
ware life  cycle  (see  Figure  5) . Since  the  main  empnasis  of 
tnis  research  stems  from  the  idea  that  much  can  be  said 
about  the  quality  and  reliability  of  software  from  the  soft- 
ware’s error  history,  only  error  data  were  analyzed.  The 
primary  approach  has  been  not  to  repeat  other  research,  but 
to  verify  and  expand  previous  findings. 


The  purpose  of  this  chapter  is  to  summarize  an  analysis 
of  data  collected  from  the  three  projects.  This  will  be  ac- 
complished by  presenting  the  results  of  an  empirical  and 
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regression  analysis  of  the  raw  data.  Chapter  VI  will 
present  the  final  empirical  aodels  developed  from  the  analy- 
sis sumaarized  in  this  chapter. 

Analysis  of  Empirical  Data 

This  section  contains  the  results  of  an  analysis  of 
software  errors  by  type.  Since  error  data  by  category  type 
was  not  available  for  Project  3,  only  error  data  from  Pro- 
jects 1 and  2 were  analyzed. 

Usinq  the  error  cateqory  list  in  Table  2,  the  question 
naturally  arises  "how  many  of  each  type  were  there?"  This 
analysis  took  place  only  at  the  major  cateqory  level,  and 
only  errors  which  required  a chanqe  to  the  source  code  or 
data  base  were  examined.  Categories  "DATA  INPUT  ERRORS"  and 
"DATA  OUTPUT  ERB03S"  were  combined  into  "INPUT/OUTPUT  ER- 
RORS". Cateqories  "DATA  HANDLING  ERRORS"  and  "ARRAY  PROC- 
ESSING ERRORS"  were  combined  into  "DATA  HANDLING  ERRORS, 
cateqories  "OPERATION  ERRORS",  "DOCUMENTATION  ERRORS"  and 
"PROGRAM  EXECUTION  ERRORS"  are  included  in  cateqory  "OTHER". 

Figure  10  shows  a percentaqe  breakdown  by  major  cateqo- 
ry for  Projects  1 and  2.  It  also  shows  a percentage  break- 
down when  Projects  1 and  2 are  combiued.  percentage  Dreak- 
downs  appear  to  be  reasonable  for  the  type  software  for  each 
project.  Project  2 is  an  cn-line  system  and  Project  1 is  a 
batch  system.  Variations  exist  in  the  percentage  oreakdowns 
because  of  the  kinds  of  operations  the  programs  are 
performing.  Project  1 programs  are  performing  mostly  data 
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input/output  and  data  transfers.  The  logic  within  the  pro- 
grams is  not  very  complicated.  On  the  other  hand  the  pro- 
grams in  Project  2 are  doing  more  computations  on  the  data 
and  the  loqic  flow  is  more  complicated. 

The  data  for  each  project  supports  the  notion  tnat  the 
distributions  of  the  types  are  application  dependent.  How- 
ever, when  the  data  from  the  two  projects  are  comoined,  the 
relative  importance  of  the  percentages  chanqes.  As  one  can 
see  the  loqic  type  has  the  highest  percentage  (24.6*  vs 
17. 9X) . It  is  suqqested  that  this  will  be  true  in  the  gen- 
eral case. 

Summary  of  Regression  Analysis 

Re<Fressio n Analysis  Concepts 

Situations  exist  where  there  are  two  or  more  variaules 
that  are  functionally  related.  The  problem  is  to  understand 
this  relationship.  This  is  not  an  easy  thinq  to  do  since 
the  relationship  may  be  a simple  or  a complex  one.  .lost 
often,  the  functional  relationship  is  extremely  complex,  or 
completely  unknown.  The  goal  is  to  obtain  a better  under- 
standing of  the  relationship  and  then  use  that  information 
for  prediction,  process  optimization  or  control. 

The  technique  usually  employed  in  these  investigations 
of  relationships  between  variables  is  known  as  regression 
analysis.  Regression  analysis  assumes  the  existence  of  a 
functional  relationship 


Y = Ffx(1),x<2),  ...  ,x  (n)  ;B  (0)  ,3  (1)  , ...,B(n)]  ♦ e. 
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where  Y is  the  response  (or  dependent)  variable, 
x{i)  (i=  1, 2 , . . . , n ) are  the  independent  variables,  3 ( "j ) 
(1*0,1,... ,n)  are  unknown  parameters,  and  e is  a random  er- 
ror component.  The  problem  consists  of  estimating  the 
unknown  parameters  in  the  above  equation. 

The  kind  of  relationship  between  Y and  the  independent 
variables  determine  the  type  of  reqression  analysis  to  per- 
form. If  the  assumption  of  linearity  appears  reasonaole 
then  linear  reqression  analysis  can  be  used.  If  linearity 
does  not  seem  reasonable  then  some  other  analysis  technique 
must  be  employed.  Since  this  research  assumes  linearity, 
linear  reqression  analysis  techniques  are  used.  The  rest  of 
this  section  briefly  discusses  linear  regression  analysis  as 
related  to  this  paper. 

If  one  wishes  to  determine  the  relationship  between  a 
sinqle  independent  variable,  say  x,  and  a single  response  or 
dependent  variable,  say  Y,  then  use  simple  linear  reqression 
techniques.  The  equation  to  predict  would  be  in  the  form 

Y = B (0)  * B (1)  x + e, 

where  "3(0)"  is  the  intercept,  "B(1)"  is  the  slope  and  "e" 
is  the  random  error  component.  The  procedure  is  to  use  tne 
method  of  least  squares  to  estimate  ”E(0)"  and  "3(1)"  [63]. 

If  one  wishes  to  determine  the  relationship  between 
many  independent  variables  and  a single  dependent  variaple 
then  use  multiple  linear  reqression  techniques.  The  equa- 


tion to  predict  would  be  in  the  form 
Y = B (0)  ♦ a (1)  x (1)  + B (2)  x (2)  ♦ 


♦ 3 (n)  x (n)  ♦ e , 


where  3(i)'s  are  the  unknown  parameters  (reqressioa  coeffi- 
cients) to  be  estimated  usinq  the  method  of  least  squares  . 

Y is  the  dependent  variable,  and  x(i)*s  are  the  known  inde- 
pendent variables. 

For  our  purpose,  the  independent  variables  are  the 
characteristic  metrics  (see  Table  4).  The  equation  would 
be,  if  all  metrics  were  included,  similar  to  the  followinq 
eq  uation 

N = B (0)  ♦ B ( 1 ) LC  + B (2)  10  E(3)CO  + B(4)DH  + 3(5)PC  + 

B (6 ) UBR  ♦ B (7)  EXIT  + B (8)  STOP  ♦ B(9)OSC  * 3(10)  CC  + B(11)T3 
+ 3(12)  N EX  + B ( 13)  N N E X ♦ B(14)FD  + B ( 1 5)  R D ♦ B(16)DD  + 

B ( 1 7 ) TD  ♦ B ( 1 8)  DR  * B(19)NCO  B(20)PAR  + B (2  1 ) N L + B(22)Rtf, 
where  B(i)*s  are  estimated  usinq  project  data  from  Tables  5-7. 

Most  orqan izations  do  not  want  to  expend  the  resources 
to  collect  data.  As  the  number  of  variables  increase  so 
does  the  collection  and  maintenance  costs  for  metric  and  er- 
ror data.  Therefore,  it  is  desirable  to  have  a minimum  num- 
ber of  metric  variables  to  collect  and  maintain.  Thus,  the 
objectives  of  the  reqression  analysis  are  to  determine  wnicn 
metric  variables  sinqly  correlated  with  the  numoer  of  errors 
and  to  determine  the  "best"  qroups  with  the  minimum  number 
of  variables  to  predict  the  number  of  errors  in  a proqram. 

The  reqression  results  presented  in  the  followinq  sec- 
tions are  summaries  of  outputs  from  the  Statistical  Analysis 
System  76  T691.  For  the  interested  reader  reference  [69] 
explains  different  techniques  for  determininq  which  vari- 
ables of  a collection  of  independent  variables  should  most 
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likely  be  included  in  a reqressicn  model.  Since  all 
techniques  (forward,  backward,  stepwise,  maximum  r-square, 
and  minimum  r-square)  were  available  in  SAS7  6 , they  were  all 
used  initially  to  screen  the  independent  variable  list.  Two 
statistics,  "r-square"  and  "F",  are  used  to  determine  if  a 
linear  relationship  exists.  "8-square"  is  the  propcttion  of 
variability  in  N that  is  explained  by  the  relationship  be- 
tween N and  the  independent  variables  (characteristics  met- 
rics) . "F"  is  a well  known  statistic  used  in  tests  of 

hypothesis. 

Simpl-e  Linear  Regression  Analysis  Results 

Each  variable  represents  a metric  which  serves  to  meas- 
ure to  some  deqree  the  number  of  program  errors.  In  order 
to  correlate  the  metric  with  the  number  of  errors  (N) , a 
sinqle  variable  linear  regression  analysis  was  performed  on 
each  metric  variable. 

A test  of  hypothesis  was  conducted  on  each  metric  to 
determine  if  its  reqressicn  on  N was  significant.  All  tests 
were  set  up  in  the  following  manner  with  a .05  level  of  sig- 
nificance 

H (0)  : B (1)  =0, 

H (1)  : B (1)  #0. 

and  under  the  assumption  that  the  e(i)'s  are  normally  dis- 
tributed. The  regression  statistics  for  each  respective 
project  is  summarized  in  Tables  11-13.  The  "Decision" 
column  indicates  if  H (0)  is  rejected  or  accepted.  Rejected 
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TABLE  11.  SUMMARY  OF  SINGLE  VARIABLE  REGRESSION  DATA 

FOR  PROJECT  CNE 


Metric  l 
Variable  | 

R-square  | 

1 

F Value  | 

1 

Pr  > F | Decision 

1 

1 

1 

1 

1 

1 

LC 

| 0.944435  | 

5C.99 

1 

0. 0057 

1 

Reject 

H (0) 

a 

10 

| 0.212239  | 

0.8  1 

J 

0. 4349 

1 

Accept 

H(0) 

CO 

1 0.627788  | 

5.06 

1 

0.  1100 

1 

Accept 

8(0) 

DH 

| 0.943303  | 

49.91 

i 

0. 0058 

1 

Reject 

H (0) 

PC 

| 0.  566996  | 

3.93 

1 

0.  1418 

1 

Accept 

H (0) 

UBR 

| 0.995852  1 

720. 16 

1 

C. 0001 

1 

Reject 

H (0) 

EXIT 

| 0.285568  | 

1.20 

1 

0. 3535 

i 

Accept 

8(0) 

il 

STOP 

| 0.056103  | 

0.  18 

1 

0.  7013 

1 

Accept 

H (0) 

OSC 

| 0.074677  | 

0.24 

1 

0. 6564 

1 

Accept. 

H (0) 

CC 

| 0.000000  | 

0.00 

1 

1. 0000 

1 

Accept 

8 (0) 

TS 

) 0.677116  | 

6.29 

1 

0.0871 

1 

Accept 

H (0) 

NEX 

J 0.945817  | 

52.37 

1 

0.0054 

1 

Accept 

a (0) 

N N EX 

| 0.232533  | 

0.91 

1 

0. 4107 

1 

Accept 

8(0) 

FD 

| 0.  1 11622  | 

C.  38 

1 

0. 5827 

J 

Accept 

H (0) 

RD 

| 0.745010  | 

8.77 

1 

0. 0595 

1 

Accept 

8 (0) 

DD 

| 0.430402  | 

2.27 

J 

0. 2292 

1 

Accept 

8(0) 

TD 

J 0.470261  | 

2.6b 

J 

0. 2012 

1 

Accept 

8 (0) 

DR 

| 0.885232  | 

23.14 

1 

0. 0171 

1 

Reject 

H (0) 

NCO 

J 0.095886  | 

0.32 

J 

0. 6121 

1 

Accept 

8 (0) 

■ 

PAR 

| 0.421938  | 

2.  19 

1 

0. 2355 

1 

Accept 

8(0) 

NL 

| 0.557331  J 

3.78 

1 

0.  1472 

J 

Accept 

8 (0) 

RW 

| 0.877502  | 

21.49 

1 

0. 0189 

1 

Reject 

8 (0) 

I 


Metric 

V ariable 

| R-square  J 

1 1 

1 1 

F Value 

Pr  > F 

J 

1 

1 

Decision 

LC 

1 1 

| 0.830952  | 

216.28 

C.0001 

1 

1 

Re-ject 

H (0) 

10 

| 0.  081709  | 

3.92 

0. 0541 

1 

Accept 

K (0) 

CO 

| 0.696377  | 

100.92 

0. 0001 

1 

Reject 

H (0) 

DH 

J 0.751509  | 

133.07 

C. GO0 1 

1 

Reject 

H (0) 

PC 

| 0.  295987  J 

13.50 

0. 0001 

1 

Re ject 

a (0) 

UBR 

J 0.821589  1 

202.62 

0. 0001 

1 

Reject 

H (0) 

EXIT 

| 0.209254  ] 

11.64 

0. 0014 

1 

Reject 

H (0) 

STOP 

i 0.  014800  | 

0.66 

0. 4206 

1 

Accept 

H (0) 

OSC 

i 0.  174450  | 

9.30 

0. 0039 

1 

Reject 

H (0) 

CC 

| 0.001323  | 

O.Qb 

C. 8103 

1 

Accept 

H (O) 

T5 

1 0.755609  | 

1 36. 04 

C. 0001 

1 

Reject 

H (0) 

NEX 

| 0.809248  | 

136.67 

0. 0001 

1 

Reject 

H (0) 

NNEX 

| 0.271785  | 

16.42 

0. 0002 

1 

Reject 

a (0) 

FD 

| 0.0  10272  | 

0.46 

0. 5027 

1 

Accept 

H (0) 

HD 

j 0.  162182  I 

3.52 

0. 0055 

1 

Accept 

3(0) 

DD 

J 0.29  1975  | 

18.14 

0. 0001 

1 

Reject 

H (0) 

TD 

| 0.  236922  | 

17.70 

C.0001 

1 

Reject 

H (0) 

DR 

| 0.799424  | 

1 75. 37 

0.  0001 

1 

Reject 

H (J) 

NCO 

J 0.  023480  | 

1.06 

0. 3093 

1 

Accept 

H (0) 

PAR 

| 0.647323  | 

80.76 

0. 0001 

1 

Reject 

a (0) 

NL 

| 0.668134  | 

83.58 

0. 0001 

1 

Reject 

3(0) 

RW 

| 0.742020  | 

126. 56 

0.0001 

1 

Reject 

H (0) 
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means  that  the  reqression  of  the  metric  on  N is  signifi- 
cant. Accepted  means  that  the  regression  is  not  signifi- 
cant. 

A summary  of  the  significance  of  the  regression  of 
single  metrics  on  N is  presented  in  Table  14.  An  means 

the  regression  is  significant.  It  is  significant  that  "LC" 
is  the  highest  ( regardless  of  software  type,  operating  mode 
or  other  project  differences)  in  Projects  2 and  3 and  second 
in  Project  1.  Variables  "DH"  and  "RW"  were  also  significant 
for  all  projects. 

Although  most  variables  turned  out  to  be  significant 
for  one  or  the  other  projects,  "LC”  and  "UBR"  are  more  im- 
portant since  these  can  be  obtained  before  coding  starts 
(they  can  be  read  directly  from  "control  flow  charts) . Tne 
predicted  linear  equations  for  all  variables  by  project  is 
contained  in  Table  15.  An  means  no  equation  was  pre- 

dicted for  this  project. 

Mu-ltiple  Linear  Reqression  Analysis  Results 

The  purpose  of  the  multiple  reqression  analysis  was  to 
determine  which  metric  variables  should  most  likely  be  in- 
cluded in  a reqression  model.  Mainly  we  were  interested  in 
screening  the  list  of  22  metric  variables  shown  in  Table  4 
to  eliminate  the  ones  that  did  not  influence  software  error 


TABLE  14. 


SUMi 1ARY  OP  SIGNIFICANCE  OF  REGRESSION 


Metric  | 
Variable  | 

FOR 

ALL 

PROJECTS 

Project 

— 

One 

1 

1 

1 

Two 

Three 

LC  | 

* 

1 

1 

* 

* 

DH  I 

* 

1 

* 

* 

CJBR  I 

* 

1 

* 

* 

DR  | 

* 

1 

* 

* 

RW  | 

* 

1 

* 

* 

CO  / 

1 

* 

♦ 

PC  | 

1 

* 

* 

EXIT  | 

1 

* 

♦ 

NEX  | 

1 

* 

* 

TS  | 

1 

* 

* 

PAR  I 

1 

* 

* 

NL  I 

1 

* 

* 

NCO  | 

1 

* 

OSC  | 

1 

* 

N NEX  | 

1 

* 

DD  | 

1 

* 

TD  | 

1 

* 

10  | 

1 

STOP  | 

1 

CC  | 

I 

FD  | 

1 

RD  ] 

1 

1 
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TABLE  15.  SINGLE  VARIABLE  MODELS  FOR  PREDICTING 
THE  NUMBER  OF  ERRORS  IN  COBOL  PROGRAMS 


Metric 

1 

1 

Proiect 

l 

One  | 

Proiect  Two 

l 

| Proiect 

Three 

Variaole 

1 

I 

1 

1 “ 
1 

Inter- | 

■ 11  J 

Slope  J 

i 

Inter-]  Slope|  Inter- | 

Slope 

I 

cept  | 

1 

cept  ] 

1 cept  | 

best  available  copy 


Analysis  Usinq  all  Metric  Variables 

The  basic  approach  to  this  part  of  the  analysis  was  to 
study  the  outputs  from  all  five  techniques  referenced  aoove 
and  select  the  sets  of  variables  most  frequently  included 
for  each  project.  Table  16  lists  a few  of  the  sets  of  vari- 
ables selected  for  all  projects.  The  hiqhest  "r-square" 
value  is  qiven  when  the  same  set  was  selected  more  than 
once.  Column  "Decision"  reflects  the  decision  relative  to 
the  hypothesis 

H (0)  : 3 (1)=B  (2)...=B(n)  = 0, 

aqainst 

H(1):  B(i)#G  for  at  least  one  i. 

and  under  the  assumption  that  the  e(i)'s  are  normally  dis- 
tributed. At  a .05  siqnificance  level,  at  least  one  B (i)  is 
siqnif icantly  different  from  zero  for  all  selected  qroups. 
Table  17  contains  the  final  list  of  variables  chosen  from 
those  shown  in  Table  16.  Table  18  contains  a summary  of  the 
predicted  equations  for  each  set  of  variables  by  project. 

The  precedinq  discussion  was  based  upon  an  objective 
analysis  of  22  variables  without  considerinq  their  relation- 
ship to  each  other.  Since  some  of  the  22  variables  were 
known  to  be  functionally  related,  i.e., 

TS  = HEX  ♦ N NEX  and  TD  = FD  ♦ HD  + CD,  a sunset  of  variaoj.es 
considered  unrelated  was  selected  and  used  in  a raqression 
analysis.  These  variables  are  LC,  10,  CO,  DH,  PC,  U3R, 

EXIT,  STOP,  OSC,  CC,  TD,  PAH,  and  DH.  The  results  from  this 
analysis  is  presented  in  the  next  section. 
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TABLE  16. 


VARIABLES  SELECTED  FOR  MODELS 


1 

No  in | Variables  Selected 
Group | 

1 

J 1 

1 R I 

| Square! 

1 1 

F | 

Value  | 

1 

Pr>F  |Deci- 
J sion 

1 

2 

1 

ILC 

OBR 

1 1 

J 0. 9585 J 

127. 11| 

1 

0.0001 13 

H (0) 

I LC 

STOP 

10.99111 

612.281 

0.0001  1R 

a (0) 

| tJ3B 

1 

EXIT 

| 0. S988J 

855.77| 

0.00  1 2 1 R 

rf  (0) 

3 

1 

JUBfi 

EXIT  NNEX 

1 1 

I 0.  99991 

999.99 | 

1 

0.00011  R 

H (0) 

| LC 

OBR  CC 

I0.9935J 

850.6  C| 

0.0001  JR 

a (0) 

|LC 

CO  STOP 

| 0. 9999 | 

611.511 

0.0001  1R 

a (0) 

I LC 

1 

CC  NEX 

I 0.99461 

1 1 

506. 4 C J 

0 . 0 0 0 1 | R 

1 

a (0) 

4 

1 

| UBS 

10  EXIT  NNEX 

1 1 
| 1. OGCOI 

999.991 

1 

0.00011  2 

H (0) 

|DD 

TD  DR  RW 

I 1 . CO  00 | 

999.99  | 

0.0001  |R 

a (0) 

| LC 

CO  UBR  STOP 

| 0.  99  1 5 1 

461.511 

0.0001  | R 

H (0) 

|LC 

EXIT  CC  NEX 

I 0.  99  49  | 

441.861 

0. CO  0 1 1 R 

H (0) 

| LC 

UBR  OSC  CC 

1 0. 8b60| 

66.20| 

0.0001  | 3 

a (0) 

| PC 

1 

STOP  CC  NEX 

| 0. 8784| 

1 1 

74.07| 

0. 0001 1 R 

1 

H (0) 

5 

1 

|UBR  EXIT  NNEX  TS  RW 

1 1 

| 1 „ CO  00  J 

649. 7 C | 

1 

0.0001  | R 

a (0) 

| LC 

CO  UBR  STOP  DR 

JO. 9955| 

354. 02 | 

0.0001 | R 

a (0) 

| LC 

CO  UBR  DR  PAR 

1 0. 9955| 

354. 02 | 

0.0001 |R 

H (0) 

|LC 

PC  EXIT  STOP  NEX 

|0„ 9975| 

6 5 2.77| 

0.0001JR 

H (0) 

|LC 

UBR  OSC  CC  FD 

| 0. 87  16 | 

54.29J 

0.0001  | R 

H (0) 

I PC 

1 

STOP  CC  NNEX  NL 

| 0. 88  08  J 

1 1 

59. 08 | 

0. 0001 1 R 

1 

8 (0) 

10 

1 

|LC 

10  CO  UBR  EXIT  STOP  DD 

1 1 

1 1 

1 

1 

TD  DR  PAR 

| 0. 9995| 
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TD 
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1 

1 
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NL 
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1 

1 

PAR  RW 
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411.44| 

0. 0002 | R 

H (0) 

|LC 

PC  STOP  NEX  FD  RD 

DD  DR 

i 1 

1 

1 

TS  RW 

|0.9999| 

2310.7J 

0.  0001 1 R 

a (0) 

|LC 
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CC 
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1 

FD  RD  TD 

| 0. 8662 | 

27.25| 
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TABLE  16.  VARIABLES  SELECTED  FOR  MODELS — Continued 


No  ia 
Group 


10 


1 

I LC 

PC 

CC  NEX 

NNEX 

RC  DD  TD 

1 1 
1 1 

1 1 

1 1 

1 

NL 

TS 

1 0. 91 351 

36. 95)0.000113 

H (0) 

| LC 

10 

CO  DH 

PC  UBR 

EXIT 

1 1 

I 1 

1 

OSC 

CC  TD 

| 0.99551 

408. 1 5 | 0. 0001|R 

a (0) 

1 LC 

10 

CO  UBR 

STOP 

OSC  CC 

1 i 

1 1 

1 

NEX 

FD  DR 

1 0.88  331 

27. 34] 0. 0001 J 3 

H (0) 

1 DH 

PC 

STOP  CC  NEX 

NNEX  FD 

1 1 

1 1 

1 

1 

RD 

TS  NL 

I 0.  8908| 

l i 

28.5510..  0001J  R 

l l 

a (0) 

15 


20 


Variables  Selected 


LC  10  CO  UBR  EXIT  CC  FD  RD 
STOP  TD  DR  NL  PAR  Rtf  DD 
LC  10  DH  PC  UBR  CC  HEX  RD 
NNEX  DD  TD  DR  NCO  TS  NL 
LC  CO  DH  PC  U3R  OSC  CC  FD 
RD  DD  TD  NCO  NL  TS  DR 


LC  10  CO  DH  PC  OBR  STOP  CC 
OSC  TS  NEX  NNEX  FD  RD 
DD  TD  DR  NCO  NL  PAR 
LC  10  CO  DH  PC  UBR  EXIT  CC 
STOP  OSC  TS  NEX  NNEX  FD 
RD  DD  TD  DR  NL  PAR 


R 

Square 


1. CCOO 
0.  92  16 
0.9186 


0. 9244 

0. 9244 


F 

Value 


99999. 
2 J.  49 
22.56 


15.28 


Pr>F 


0.0001 
0.0001 
0. 0001 


0.0001 


Deci- 

sion 


a h (0) 

8 H (0) 
R H (0) 


R K (0) 


1 3.  9810.0001  JR  H (0) 
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TABLE 

17. 

FINAL  LISTS 

OF  METRIC  VARIABLES  SELECTED 

Metric 

Variable 

1 

1 

1 

1 

Number  of 
1 

1 1 

Variables  in 

1 i 

J 

models 

1 

1 

1 

2 I 5 

1 10  | 

15  1 20 

LC 

1 

* 

1 

* 

1 

* 

1 

* 

l 

* 

UBR 

1 

* 

1 

* 

1 

♦ 

1 

* 

1 

* 

10 

1 

1 

* 

I 

* 

1 

* 

1 

* 

DH 

J 

1 

* 

1 

* 

1 

* 

J 

* 

EXIT 

1 

i 

* 

1 

* 

1 

* 

1 

* 

CO 

1 

1 

1 

* 

1 

* 

1 

* 

PC 

1 

1 

1 

* 

1 

* 

1 

* 

OSC 

1 

1 

1 

* 

1 

* 

1 

♦ 

CC 

1 

J 

J 

* 

* 

1 

* 

STOP 

1 

1 

1 

» 

* 

1 

* 

TD 

1 

1 

1 

* 

t 

* 

1 

* 

DR 

1 

1 

1 

k 

* 

1 
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PAR 

1 

1 

1 

1 

* 

1 

* 

NL 
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1 
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J 

* 

1 

• 

RW 

1 

1 

1 

1 

* 

1 

* 

TS 

1 

1 

1 

1 

1 

* 

NNEX 

1 

1 

1 

1 

1 

* 

FD 

1 

1 

t 

1 

1 

* 

DD 

1 

1 

1 

1 

1 

* 

NCO 

1 

1 

J 

1 

1 

* 

NEX 

1 

1 

1 

» 

1 

RD 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

i 


0.047EXIT  - 1.449ST0P  - O.2710SC  - 2.589CC  + 0.324TS  - 0.336NNEX 
0.065FD  - 0.078DD  + 0.094TD  + 0.002DR  + O.O12NC0  - 0.044PAR 
0 - 011NL  + O.OOIRW 


I 

I 

I 

I 

I 
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Analysis  of  Seduced  Sets  of  Unrelated  Variables 

It  is  appropriate  at  this  time  to  point  out  that  analy- 
sis was  not  performed  for  a project  containing  a number  of 
observations  less  than  the  number  of  metric  variables. 


All  tests  of  hypothesis  were  set  up  in  the  following 
manner  with  a 0.05  level  of  significance 
H (0)  : B (1)  =B  (2)  =. . .-=B  (n)  = 0 

against 

Ei  ( 1 } : B (i)  * 0,  for  at  least  one  i, 
and  under  the  assumption  that  the  e(i)*s  are  normally  dis- 
tributed. The  results  of  this  analysis  are  summarized  in 
Table  19.  The  regression  is  significant.  Therefore,  this 
final  set  of  13  metrics  is  a good  predictor  of  software  er- 
rors. 

Conclusions 

The  main  purpose  of  the  preceding  analysis  was  to  de- 
termine if  program  characteristics  metrics,  whicn  measure 
program  complexity,  are  predictors  of  the  number  of  errors 
in  COBOL  programs.  It  was  shown,  through  simple  linear  and 
multiple  linear  regression  analysis,  that  the  number  of  er- 
rors in  a COBOL  program  is  a function  of  its  structure  wnicn 
is  measured  by  characteristics  metrics. 

A set  of  13  unrelated  metrics  was  chosen  as  the  final 
group  of  metrics  to  predict  the  number  of  software  errors. 
The  next  chapter  will  specifically  discuss  how  this  set  is 
used  as  a measure  of  program  complexity. 


i 


I 

I 


VI 


A MEASURE  OF  PROGRAM  COMPLEXITY 


Introduction 

Initially  it  was  hypothesized  that  the  number  or  soft- 
ware errors  could  be  predicted  from  the  internal  complexity 

of  the  proqrams.  But  what  does  one  mean  by  "internal  corn- 
's 

plexity"?  Is  it  a property  that  can  be  observed  and  meas- 
ured, and  perhaps  even  related  to  the  uuaber  or  errors  in 
proqrams? 

Complexity  of  any  object  is  some  measure  of  the  mental 
effort  r 4,171  required  to  understand  that  object.  In  gener- 
al usaqe,  the  complexity  of  an  object  is  a function  of  tne 
relationships  among  the  components  of  the  object.  As  ap- 
plied to  computer  proqrams,  it  is  a measure  of  tne  internal 
structural  characteristics  of  the  proqram.  The  previous 
chapter  presented  the  results  of  an  analysis  that  showed 
that  actual  proqram  characteristics  were  related  to  the  num- 
ber of  proqram  errors.  The  purpose  of  this  chapter  is  to 
define  a "program  complexity  measure"  from  these  basic  char- 
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other  external  programs  such  as  the  operating  system  and 
other  application  programs,  program  logic,  etc.  These  com- 
ponents and  the  relationships  among  them  determine  program 
complexity.  The  problem  is  to  measure  the  degree  to  which 
certain  relationships  exist  within  a program.  For  example, 
the  number  of  input/output  statements  measures,  in  some 
degree,  whether  an  input/output  relation  exists  between  tne 
program  and  a data  base.  Complexity  when  applied  to  a spe- 
cific relation  is  called  local  complexity.  The  complexity 
of  a program  as  a whole  is  defined  in  terms  of  local  com- 
plexities. There  are  7 local  complexities:  control  flow 
complexity,  input/output  complexity,  data  use  complexity, 
computational  complexity,  data  transfer  complexity,  struc- 
ture design  complexity,  and  interface  complexity. 

Control  flow  complexity  is  defined  as  the  number  of 
logical  relationships  present  in  the  source  code.  In  a 
COBOL  proqram,  these  relations  are  manifested  as  IF,  GOTO, 
STOP,  and  PERFORM. .. UNTIL  statements,  and  AND  and  OR  condi- 
tions. The  Control  Flow  Complexity  metric,  CFC,  can  oe 
numerically  evaluated  for  each  program  by  calculation: 

CFC  = LC  ♦ UBR  + STOP 

A Normalized  Control  Flow  metric,  NCFC,  is  defined  as  NCFC  = 
CPC/1000. 

Input/output  complexity  is  defined  as  the  number  of  I/O 
statements.  The  Input/Output  Complexity  metric,  aobreviatea 


i 


1 


as  IOC,  is 


IOC 


10 


8 3 


Data  use  complexity  is  defined  as  the  ratio  of  data 
references  (actual  references  to  data  items)  to  total  data 
definitions.  Many  data  reference  errors  are  made  because 
the  assumptions  made  when  defininq  data  differs  from  the  as- 
sumptions made  when  usinq  the  data.  For  example,  when 
defininq  a data  item  as  ALPHABETIC,  it  is  assumed  that  only 
alphabetic  data  will  be  stored;  but,  when  numeric  data  is 
stored  in  the  data  item  it  is  assumed  that  the  data  item  is 
declared  "numeric"  or  "alphanumeric".  As  the  number  or  data 
references  increases  relative  to  total  definitions,  the  data 
use  complexity  increases.  The  Data  Use  Complexity  metric, 
abbreviated  as  DUC,  is  calculated  as 

DUC  = DR/TD. 

Computational  complexity  is  defined  as  the  number  of 
arithmetic  statements  such  as  ADD,  MULTIPLY,  COMPUTE,  etc. 
The  metric  for  this  complexity  is  aobreviated  as  COC  wnere 

COC  = CO. 

Data  transfer  complexity  is  measured  by  the  number  of 
data  transfer  statements.  This  complexity  metric, 
abbreviated  as  DHC,  is  evaluated  by  countinq  the  number  of 
data  transfer  statements  (DHC  = DH) . 

Interface  complexity  is  measured  by  the  number  of  sys- 
tem interfaces  (number  of  system  routines  called)  and  the 
number  of  application  routine  interfaces  (number  of  internal 
and  external  application  routines  called) . The  Interface 
Complexity  metric,  abbreviated  as  IC,  is  defined  as  rollows: 

IC  = OSC  ♦ CC  ♦ PC, 
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where 


OSC  = number  of  system  program  interfaces 
CC  = number  of  compiler  calls  such  as  COPY 
PC  = number  of  applicaticn  (internal  and  external) 
routine  interfaces. 

Structural  desiqn  complexity  is  a measure  of  the  number 
of  distinct  routines  (number  of  components)  in  a proqram. 

The  structural  Desiqn  Complexity  metric  referred  to  as  SC, 
is  numerically  evaluated  for  each  proqraa  by  calculating: 

SC  = (PAH  - EXIT)  +1. 

The  1 accounts  for  the  main  control  flow  routine. 

Total  complexity  is  a function  of  the  7 local  complexi- 
ties discussed  above.  The  Total  Complexity  metric,  referred 
to  as  TC,  is  defined  as  fellows: 

TC  = CPC  ♦ IOC  * DOC  COC  + DHC  ♦ IC  + SC. 

A Normalized  Total  Complexity  metric,  referred  to  as  NIC,  is 
defined  as  TC  = TC/1000. 


Regression  Analysis  of  Complexity  Metrics 
Heretofore,  data  from  3 projects  were  used.  But  here,  the 
complexity  metrics  are  analyzed  using  the  oriqinal  3 data 
bases  plus  1 data  base  consisting  of  all  3 projects'  data 
combined.  The  purpose  for  mixinq  the  sources  of  data  is  to 
observe  what  happens  when  different  types  or  proqrams,  de- 
veloped by  different  organizations,  are  mixed.  Hereafter, 
the  combined  data  base  is  referred  to  as  Project  4.  There 
are  65  observations  (programs)  in  this  data  base.  Taoles 


I 

I 


1 
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20-23  summarize  the  reqression  statistics  for  each  respec- 
tive project.  The  "Decision"  column  reflects  the  decision 
at  the  .05  siqnificance  level  relative  to  the  hypothesis 


aqainst 


H (0)  : B =0, 


H (1) : B * 0 , 

and  under  the  assumption  that  the  e (i ) *s  are  normally  dis- 
tributed. Table  24  presents  a summary  of  the  reqression 
siqificance,  at  the  .05  level,  of  the  complexity  metrics  oy 
project. 

The  "best"  complexity  models,  predicted  by  the  Maximum 
B-square  technique,  are  listed  by  project  in  Table  25.  The 
aodels  for  total  complexity  and  the  normalized  metrics,  TC, 
NTC  and  NCFC,  are  listed  in  Table  26.  Actual  data  points 
(plotted  as  '+*)  and  the  reqression  line  for  project  3 are 
shown  in  Fiqures  11-20. 


Conclusions 

The  purpose  of  this  chapter  was  to  define  and  develop  a 
"proqraa  complexity  measure"  from  13  unrelated  characteris- 
tics metrics  selected  from  the  analysis  presented  in  the 
previous  chapter.  Seven  local  complexities  were  defined  and 
used  to  develop  a measure  of  total  proqraa  complexity.  Met- 
rics for  each  complexity  were  defined  also.  It  was  shown, 
throuqh  simple  linear  and  multiple  linear  reqression  analy- 
sis, that  the  number  of  errors  in  COBOL  proqrams  is  a func- 
tion of  these  complexity  metrics. 


Linear  models  developed  from  these  metrics  can  be  used 
to  predict  the  number  of  errors  in  COBOL  programs.  The 
"best"  single  variable  model  for  predicting  errors  is  the 
Control  Plow  Complexity  metric  model.  fhe  "best"  multiple 
variable  model  for  predicting  errors  is  the  one  that  con- 
tains all  7 local  complexity  metrics.  Analysis  of  Project  h 
showed  that  the  latter  model  can  be  used  when  dealing  with 
many  types  of  programs  that  are  developed  by  different  or- 
ganizations. However,  it  is  suggested  that  each  organiza- 
tion estimate  the  model  parameters  relative  to  error  data 
from  its  development  projects. 
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TABLE  22-  SUMMARY  OF  THE  COMPLEXITY  METRICS  REGRESSION 

DATA  FOR  PROJECT  3 
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TABLE  24.  3 EGRESSION  SIGNIFICANCE  OF  COMPLEXITY  METRICS 

VERSUS  NUMBER  OF  ERRORS 
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TABLE  26.  TOTAL  AND  NORMALIZED  COMPLEXITY  MODELS  BY  PROJECT 


PROJECT 


MODELS 


1 


N = - 3.728  + 0.153TC 


N = - 3.728  + 153. 060NTC 
N = 0.287  + 381 .860NCFC 


2 


N = 1.374  + 0.042TC 
N = 1.374  + 41. 977NTC 
N = 8.943  + 95. 763NCFC 


3 


N = 2.220  + 0.011TC 
N = 2.220  + 11. 222NTC 
N = 4.069  + 23.830NCFC 


N = 7.489  + 0.012TC 

N = 7.489  + 11.666NTC 
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Figure  12.  Plot  of  I/O  Complexity  Model  for  Project  3 
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Figure  18.  Plot  of  the  Total  Program  Complexity  Model  for  Project  3 
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SOildABY,  CONCLUSIONS  AND  B3C0.1UEN DATIONS 
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Suamary 

The  high  incidence  of  errors  in  software  is  the 
underlying  prodera  of  software  reliaaility.  But,  the  most 
important  unknown  of  software  reliability  is  the  number  of 
residual  errors  in  a proqram.  If  this  nuaoer  were  available 
early  in  the  software  development  staces,  the  software  engi- 
neering process  would  be  enhanced  greatly.  Several  other 
unknowns  could  then  be  solved.  One  could  determine  when  to 
stop  testing  a proqram,  estimate  the  cost  of  maintenance  and 
establish  levels  of  confidence  in  programs  ana  systems  of 
proqrams,  and  develop  more  accurate  software  models  that 
would  model  software  failures  more  realistically.  The 
results  would  be  the  ability  to  deal  with  all  the  unknowns 
of  software  reliability  and  reduce  the  overall  cost  of  soft- 
ware. 

The  goals  of  this  research  were  to  determine  if  actual 
program  characteristics  are  predictors  of  the  number  of  er- 
rors in  COBOL  proqrams,  to  define  proqram  complexity  meas- 
ures from  proqram  characteristics,  and  to  propose  complexity 
models  for  predicting  the  number  of  errors  in  a COBOL  pro- 
qram. 
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Throuqh  simple  linear  and  multiple  linear  regression 
analysis  of  3 sources  of  data,  the  number  of  errors  in  a 
COBOL  proqram  was  shown  to  be  a function  of  its  structure 
which  can  be  measured  by  22  character istics  metrics.  A set 
of  13  unrelated  characteristics  metrics  was  selected  from 
the  22  characteristics  metrics  to  define  7 local  complexity 
metrics.  The  7 local  complexity  metrics  were  predictors  of 
the  number  of  errors  in  a proqram.  Consequently,  these  met- 
rics were  used  to  estimate  models  to  predict  errors.  The 
"best"  sinqle  variable  model  for  predicting  errors  is  the 
Control  Flow  Complexity  metric  model.  The  "oest"  multiple 
variable  model  for  predicting  errors  is  the  one  that  con- 
tains all  7 local  complexity  metrics.  Analysis  of  Project  4 
showed  that  the  latter  model  can  be  used  when  dealing  with 
many  types  of  programs  that  are  developed  oy  different  or- 
ganizations. However,  each  organization  should  estimate  the 
model  parameters  relative  to  error  data  from  its  development 
projects.  Results  relative  to  Projects  1,  2,  3 ana  4 are 
summarized  in  Appendices  B,  C,  D and  E respecti vely. 

There  are  several  applications  for  the  complexity 
models.  Some  of  them  are: 

1)  Estimating  the  number  of  errors  in  programs, 

2)  Controlling  the  quality  and  structural  complexity  of 
programs  durinq  design  [18  1, 

3)  Estimating  and  allocating  resources  for  proqram  main- 
tenance f 15,62  1, 

4)  Estimating  a level  of  confidence  in  a proqram  [ 4 ], 
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5)  Developinq  failure,  density,  and  reliability  functions 
for  software  reliability,  and 

6)  Establishing  a cut-off  point  for  debuqqinq  and 
testinq  computer  software  f 10]. 

Reqardless  of  application,  however,  it  is  necessary  to  es- 
timate the  number  of  errors.  Once  this  nujDer  is  avdilaola, 
the  problems  of  software  reliability  can  be  treated  more  ef- 
fectively. In  order  to  illustrate  the  application  of  tne 
complexity  model  in  determininq  software  reliability.  Appen- 
dix F presents  an  example  calculation  for  Project  2. 

Conclusions 

A detailed  look  at  error  types  showed  that  loqic  and 
data  handlinq  were,  percentaqewise , the  most  frequent  errors 
in  Projects  2 and  1 respectively-  However,  when  error  data 
from  both  projects  were  combined,  loqic  errors  were  the  most 
frequent  errors.  It  seems  that  the  percentaqe  of  error 
types  will  vary  dependinq  on  type  of  software;  put,  in  qen- 
eral,  loqic  errors  will  normally  oe  the  most  frequent. 

Twenty-two  proqram  characteristics  metrics  were 
analyzed  by  reqression  analysis  techniques.  Both  sinqle  and 
multiple  variable  reqression  analysis  showed  that  the  rela- 
tionship between  the  metrics  and  the  number  of  errors  was 
siqnificant.  Both  sinqle  and  qroups  of  the  structural  char- 
acteristics metrics  were  qood  predictors  of  the  number  of 
errors.  The  number  of  loqical  conditions  is  the  "best" 


sinqle  predictor 
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the  "second  best"  sinqle  predictor.  Different  como inatior.s 
of  metrics  were  qood  predictors  also.  Tne  metrics  in  the 
"best"  equations,  as  determined  by  the  Maximum  R-square 
technique,  varied  by  project.  However,  "LC"  and  "UBR"  con- 
sistently appeared  in  the  "best"  equations.  Thirteen  unre- 
lated metrics  were  selected  to  measure  proqram  coaolexity. 

Total  proqram  complexity  is  measured  by  7 local  com- 
plexity metrics.  Several  complexity  models  are  qood  predic- 
tors of  the  number  of  errors  in  COBOL  proqrams.  The  "best" 
sinqle  variable  model  for  predicting  errors  is  the  Control 
Flow  Complexity  metric  model.  The  "best"  multiple  variable 
model  for  predictinq  errors  is  the  one  that  contains  all  7 
local  complexity  metrics.  The  latter  model  can  oe  used  when 
dealing  with  many  types  of  proqrams  that  are  developed  by 
different  orqan izations. 


Re co mj e ndations 

One  very  worthwhile  outcome  of  this  study  was  a posi- 
tive attitude  toward  beinq  able  to  predict  software  errors 
from  complexity  measures.  This  paper  only  scratched  the 
surface  by  showing  that  proqram  complexity  could  be  used  to 
predict  the  number  of  errors  in  COBOL  proqrams.  However, 
the  measures  should  apply  to  all  languages.  Other  research 
areas  are  discussed  below. 

The  results  from  the  error  type  analysis  indicate  that 
error  types  did  have  some  sort  of  distribution.  Complexity 


I 


I 


I. 


109 

metrics  for  specific  error  types  would  be  very  useful  for 
costinq  and  scheduling  proqram  maintenance. 

The  Data  rJse  metric  seems  promising.  It  seems  that  tae 
Reserve  Word  characteristic  metric  is  related  to  this  met- 
ric. More  research  is  needed  to  determine  if  this  is  true. 

It  was  shown  that  the  number  or  errors  are  predictable 
from  complexity  measures.  We  hypothesize  that  the  number  of 
personnel  assiqned  to  a development  project,  total  software 
cost,  total  development  time,  computer  test  time,  mainte- 
nance cost,  and  program  enhancement  cost  are  also  functions 
of  proqram  complexity  measures.  Additional  research  is 
needed  to  determine  if  this  aypothesis  is  true. 
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APPENDIX  A 


A brief  summary  of  the  more  important  concepts  associ- 
ated with  the  underlying  mathematical  reliability  theory  as 
applied  to  hardware  is  presented  for  those  readers 
unfamiliar  with  reliability. 

If  a aore  detailed  insight  is  desired  then  the  reader 
should  seek  other  references  such  as  £69-71], 


The  reliability  of  a component  is  defined  as  the  proba- 
bility that  the  component  will  function  within  specified 
liaits  for  a specified  period  of  time  under  specified  envi- 
ronmental conditions.  The  frequencies  at  which  components 
fail  per  unit  time  is  called  failure  rate.  Its  reciprocal 
value  is  called  mean- tiae-to-f ailure,  abbreviated  as  MTTF. 
Several  probability  distributions  are  employed  in  the  study 
of  reliability. 


Tiae-to-Failure  Distribution 
Let  f (t)  be  the  probability  density  of  the  tiae  to 
failure  or  malfunction  of  a component;  that  is,  the  proba- 
bility that  the  coaponent  will  fail  between  tiaes  t and 
(t>At)  is  given  by  f(t).At.  The  probability  that  the  compo 
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nent  will  fail  sometimes  before  t is  given  by 


t 

F(t)  = / f(t)dt 
o 

which  is  sonetioes  called  the  MunreliabilityM  function. 

Beliabilitv  Function 

The  probability  that  a conponent  will  survive  to  time  t 
is  qiven  by  the  reliability  function 

R (t)  * 1 - P (t ) . 

The  reader  should  note  the  relationships  between  f(t),  F (t) 
and  R (t) . In  particular 

f (t)  = df/dt  = -dB/dt. 

Bean-Tine-To- Failure 

A measure  of  effectiveness  often  required  in  reliabili- 
ty is  the  BTTF.  This  is  found  by  talcing  the  first  moment  of 
the  mean  of  the  time  to  failure  distribution-  In  terns  of 
the  density  f { t ) , 


MTTF  = / tf(t)dt. 
o 

An  equivalent  expression  giving  the  MTTF  in  terms  of  the  re- 
liability function  is 


MTTF  = / R(t)dt. 
o 
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Instantaneous  Failure  gate 

The  probability  that  a component  will  fail  in  the  in- 
terval from  t to  t*At,  qiven  that  it  has  survived  to  time  t, 
is  as  follows: 


P(t<T<  t+  At|T  > t)  = 


F(t  -I-  At)  - F(t) 
R(t) 


dividing  this  expression  by  At  yields  an  average  rate  of 
failure  in  the  interval  from  t to  t*-  At,  given  that  it  has 
survived  to  time  t,  as  follows: 


F(t  + At)  - F(t) 
At 


rttt 


By  talcing  the  limit  of  the  last  expression  as  At  -+■  0,  the 
instantaneous  failure  rate  or  hazard  function  H (t)  is  ob- 
tained; that  is. 


H(t)  = 


11m 

F(t+At-F(t) 

1 = 

(i till] 

i 

At-K) 

At 

* R(t) 

R(t) 

using  the  identities  involving  f ( t)  , F (t)  and  R (t)  we  get 
the  following  equivalent  expressions  for  3 (t)  : 

H(t)  = f(t)/R(t) 

- dR(t)/dt 
R(t) 


- In  [R(t)]< 
dt 
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This  differential  equation  is  solved  for  H (t)  to  yield 

t 

R(t)  = exp  (-  / H(t)dt)  , 
o 

and  since  H (t)  = f(t)/R(t)  we  get 

f(t)  = H(t)  exp  (-  /t  H(t)dt)  (Al) 

o 

Expression  (Al)  shows  that  the  tine  tc  failure  density  is 
related  to  the  instantaneous  failure  rate  function.  Also 
(Al)  is  a qeneral  expression  that  applies  to  any  type  of 
failure  density  and  hazard  rate  functions.  Figure  21  shows 
a typical  hazard  function  as  a function  of  age. 


The  Exponential  Model 

There  are  situations  where  a component  reaches  a point 
in  its  life  cycle  where  the  failure  rate  is  constant,  that 
is, 

H (t)  * c,  c > 0. 

On  substituting  into  equation  Al  we  get  the  time-to-f ailuce 
density 

f (t)  = cexp(-ct),  t > 0 

which  is  the  exponential  probability  density  function,  see 
Fiqure  22.  Further  calculations  show  that 

H (t)  - exp  (-ct) 


and 


HTTF  = 1/c 


COMPONENT  FAILURE  RATE  AS  A FUNCTION  OF  AGE 
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The  exponential  model  is  the  most  widely  applied  aodel 
in  reliability  enqineerinq.  This  is  due  to  its  simplicity 
and  its  theoretical  properties  such  as  constant  failure  rate 
and  loss  of  aeaory  property  [70,71],  In  many  situations 
failures  are  described  quite  well  by  the  exponential  model, 
but  there  are  also  many  examples  where  it  is  not  appropri- 
ate. 


The  Beibull  Model 

The  exponential  distribution  is  a single  parameter  dis- 
tribution which  can  be  represented  as  a special  case  of  a 
more  qeneral  two-parameter  distribution  called  the  Beibull 
distribution. 

The  assumption  of  a constant  failure  rate  is  often  ap- 
propriate for  describing  chance  failures,  but  it  is  not 
always  sufficient.  This  is  particularly  true  during  the 
early  "burn-in"  period  and  the  late  "wear-out"  period  in  the 
life  cycle  of  a component,  see  Figure  21,  Nor  would  the 
constant  failure  rate  be  appropriate  during  a period  of  re- 
liability growth  due  to  improvements  in  the  component. 

Thus,  it  is  obvious  that  a function  that  allows  an  increas- 
ing or  decreasing  failure  rate  is  required.  The  versatile 
Veibull  function  is  often  used  to  approximate  such  failure 
rates.  The  hazard  function  is 

B-l 


H ( t ) = xBt 


t > o 


XB  > o . 
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When  B < 1 the  failure  rate  decreases  with  tine;  if  B > 1 it 
increases  with  tine;  and  if  a = 1 the  failure  rate  is  con- 
stant, see  Piqure  23.  The  distribution  and  density  func- 
tions are 

f(t)  = xBtB_1  exp(-xtB),  t > o 

F ( t ) = / xBtB_1  exp(-xtB),  t > o . 
o 


The  reliability  function  is 

R(t)  = exp  (-XtB)  . 


The  Weibull  sodel  enjoys  widespread  use  because  it  can  be 
justified  theoretically  and  because  it  is  so  versatile. 


There  are 
for  describing 
normal,  qanna. 


Others  Models 

other  probability  functions  which  are  useful 
the  random  nature  of  failures.  They  are  the 
and  lognornal. 


NORMAL: 


2 

. Irina 

f(t)  = — r=  e 

a /2it 


, - 00<t<  + °° 


i r mi  i irtii  liifUtMi ilHlii 
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and 


GAMMA: 


f(t)  = t0"1.  exp(-t/B)  . 

T(a)Ba 


t,  a,  B > 0 


and 


LOGNORMAL:  f(t)  = —=L — exp[-(ln  t-a)2/2B2] 

V2tt  Bt 


B > o . 


For  appropriate  choices  of  parameters,  the  gaooa  and 
loqnoraal  functions  can  be  made  to  represent  increasing  or 
decreasing  failure  rates.  The  normal  function  is  priaarily 
used  during  the  wearout  period  of  a component,  see  Figure 
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APPENDIX  B 


Summary  of  Project  1 


Project  1 is  a data  collection  system.  The  system  pro- 
vides an  on-qoinq  data  base  for  input  into  reliaoility 
models.  The  data  base  also  contains  proqram  characteristics 
as  discussed  in  this  paper.  The  system  applies  to  COBOL 
proqrams  desiqned  to  execute  on  the  Honeywell  H6060  computer 
system  throuqhout  the  Air  Force.  The  5 proqrams  in  this 
system  utilize  a file  manaqement  system  availaDle  on  the 
H6060. 


Air  Force  Data  System  Desiqn  Center;  Gunter  AFS,  flontgomery, 
Ala. 

Computer  System:  Honeywell  H6060. 

Operating  Mode;  Batch. 


Language:  COBOL 

Total  -Number  of  Lines  of  Source  Code:  2280. 

Best  Single  Variable  Characteristics  Metric  Model: 
N = -1.683  ♦ 1.047UBR. 


N = 0.  287  ♦ 0. 382CFC. 
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APPENDIX  C 


gSgg£i£^igfll 

Project  2 is  an  on-line  system  involvinq  several  kinds 
of  data  processing  activities  such  as  personnel  management, 
accounting  and  finance,  inventory  etc-  Only  14  programs  are 
available  for  analysis. 

De-veloeeen-t-  Agency : 

City  of  Montgomery  Housing  Authority,  Montgomery,  Ala. 
Computer  System:  National  Cash  Register  NCH8200. 

Operating  Mode;  On-line. 

Nu-ebep  o-f  Programs:  14. 


La-nguage;  COBOL. 

latal-jiaabeJL-ai  Unes..Q5_Sguj[r<?e  Code,; 

N = 3.731  ♦ 0.319LC. 
figSl-.5iBgig-Ia£i£ble_Com£ie)u.ty,.flo4el 


N = 8.943  ♦ 0.096CFC. 


19045. 


N = -1.291  ♦ 0.079CFC  ♦ 0.019IOC  ♦ 0.314DUC  ♦ 0.208COC 
+ 0.005DHC  ♦ 0.056IC  - 0-074 SC 


APPENDIX  D 


:haracteristics 


Description; 


Project  3 represents  an  initial  delivery  of  a large  on- 
line Command  Manpower  Data  S ystem (CMD5) . CMOS  is  a resource 
accountinq  and  management  information  system  which  supports 
the  Manpower  and  Organization  function  at  Major  Command 
level  throughout  the  Air  Force.  Tne  programs  perform  a wide 
variety  of  data  processing  activities,  general  purpose  util- 
ity, data  retrieval,  data  maintenance,  etc.  The  programs 
utilize  a file  management  system  available  on  the  H6060. 
Development  Agency; 

Air  Force  Data  System  Desiqn  Center;  Gunter  AFS,  Montgomery, 
Alabama. 

Computer  System;  Honeywell  H6060. 

Operating  Mode:  On-line  and  batch. 


Languages:  COBOL,  PORTRAN,  and  Assembler  (only  C030L 

proqrams  analyzed  but  CALLS  to,  and  interface  errors  with, 
FORTRAN  and  assembly  language  programs  were  counted). 


This  Project  is  a combination  of  Projects  1,  2 and  3. 

Air  Force  Data  Systea  Desiqn  Center;  Gunter  AFS,  Montgomery , 
Alapaaa  and  the  City  of  Montgomery  Housing  Authority. 
Computer  Systems:  H6060  and  NCH8200. 


Operating  Modes:  On-line  and  Batch. 


APPENDIX  F 


A-pplication  to  Software  Beliabili tv 
Several  models  for  predicting  the  number  of  errors  in 
COBOL  programs  are  presented  in  this  paper-  The  Mbest"  mul- 
tiple variable  complexity  model  for  Project  2 is  used  in 
this  appendix.  The  eguation  is 

N = -1.291  ♦ 0.079CFC  ♦ 0.019IOC  ♦ 0.314DUC  ♦ 0.208COC 
♦ 0.005DHC  + 0.056IC  - 0.074SC,  (FI) 

for  the  range  of  source  data  values  contained  in  Table  6. 

But,  how  does  eguation  (FI)  apply  to  software  reliability? 
Basically,  this  eguation  offers  a solution  to  the  primary 
problem  in  software  reliability,  that  is  predicting  the  num- 
ber of  errors  in  a program.  Once  this  number  is  known,  the 
nazard,  time-to-f ailure  distribution,  and  reliability  func- 
tions can  be  derived.  Also,  the  failure  frequency  and 
mean-tine- to-failure  can  be  calculated,  and  stopping  points 
for  testing  programs  can  be  established.  To  demonstrate  how 
this  is  done,  the  Jelinski  and  Moranda  (JJ1)  [42]  model  (the 
hazard  function  was  discussed  in  Chapter  III)  is  used.  The 
basic  assuations  of  the  JN  model  are: 

1)  The  amount  of  debugging  time  between  error  occurrences 
has  an  exponential  distribution  with  an  error  occurrence 
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rate  or  hazard  function  proportional  to  the  number  of 
errors  remaining. 

2)  Each  error  discovered  is  immediately  removed,  thus 
decreasing  the  total  number  of  errors  by  one. 

3>  The  failure  rate  between  errors  is  constant. 

The  hazard  function  is 

Hf  t (i)  1 = Kf  N - (i  -1)  ] - 


= Kf  N-n  1 


where 


N is  the  total  number  of  initial  errors  in  a program, 

K is  the  proportionality  constant, 

t(i)  is  the  i-th  time  debugging  interval,  i.e. , the  time 

between  the  i-th  and  the  (i-l)-st  errors  discovered,  and 
n is  the  total  number  of  errors  found  to  date. 

As  was  pointed  out  in  Chapter  III,  the  critical  problem 
is  to  estimate  N and  K.  Eguation  (FI)  is  used  to  estimate  N 
and  the  following  eguation  [42]  is  used  to  estimate  k: 


NT  - 1 (i  -1)  t(i) 
i=  1 


where 


n is  the  numer  of  errors  found  to  date 


T * T t (i)  is  the  total  test  time  from  start  of  testing. 
i=  1 
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Proa  this  one  can  obtain  the  time-to- failure  distribution 
(density  function) 

f(t)  = K (N  - n)  exp  r-K  (N  - n)  t (i)  ].  (F4) 

The  reliability  function  is 

art(i)  1 = exp  r-K  (N  -n)t(i)  ].  (F5) 

The  aean-time-to-failure  (HTTP)  is 


1 

MTTF  = — — - . lFb 

K (N  - n) 

Example  Calculations 

Program  number  (observation)  4 from  Project  2 is  used 
to  illustrate  the  calculations.  The  observed  number  is  22 
(see  Tanle  6).  It  is  assumed  that  5 errors  have  Deen 
detected  during  8 days  of  testiuq.  Therefore,  Mn"  is  5 and 
"T"  is  8 time  units.  Each  time  unit  is  1 day.  The  proce- 
dure for  calculating  reliability  equations  is: 

1)  Calculate  N using  equation  (FI), 

2)  Calculate  K using  equation  (F2) , and 

3)  Calculate  reliability  statistics  using  reliaaility 

equations  (F4)  , (F5)  , and  (F6)  . 


1)  Por  N where  (see  Table  6) 


CFC  = LC  ♦ UBB  ♦ STOP  = 59  ♦ 59  + 1 = 119 

IOC  = 10  = 57 

DUC  = DR/TD  = 6.3474 


COC  = CO  = 53 
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DHC  = 

DH  = 

124 

IC  = 

OSC  ♦ 

CC  ♦ PC  = 

0 ♦ 0 ♦ 

18 

SC  = 

(PAH  ■ 

- EXIT)  ♦ 

1 = (22  - 

9) 

18 

1 = 14 


N = -1.291  ♦ 0.079  (119)  ♦ 0.019  (57)  ♦ 0.314  (6.3474) 

♦ 0.208  (53)  ♦ 0.005  (124)  ♦ 0.056  (18)  - 0.074  (14) 
= 22-8 


= 23  since  N is  an  integer. 
2)  For  K where 

n = 5 , T = 8 and  t (5)  = 1 

n 5 


_ n 

NT  -l  (i  - 1)  t (i)  (23)  (8)-[0*1+2  + J«-4] 

i=  1 


5 5 

0.0287/day. 

184  - 10  174 

3)  The  hazard  function  is 

H ( t)  = K (N  - n)  = 0.0287  (23  - n)  . 

A failure  curve  for  different  values  of  n is  shown  in  Figure 

24. 

4>  The  density  function  is 

frt(i)  1 = K ( M - n)  exp  [ — K ( N - n)  t(i)  ]. 

* 0.0287  (23  - n)exp[-0. 0287(23  - n)t(i)]. 

5)  The  reliability  function  is 

Hf  t (i)  1 = exp  f-0.0287  (23  - n)  t (i)  ]. 

Beliability  curves  for  different  values  of  n are  plotted  in 
Fiqure  25.  A few  calculations  were  t (i)  = 1 and  n varies 
are  shown  below: 


Figure 
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I 

I 

I 

I 


I 


I 


a. 


If  t{i)  = 1 and  n = 5 then 

a ( 1 ) = expT-O. 0287  (23-5)  1 ] 

= expf-0.0287  (18)  1 1 
= 0.5966. 

If  t(i)  =1  and  n=  10  then 

R ( 1 ) = expr-0.0287  (23-10)  1 1 
= expf-0.0287  (13)  1 1 
= 0.6886. 

If  t (1)  = 1 and  n = 22  then 

H ( 1)  = expr-0.0287  (23-22)  1 ] 
= expf-0.0287  (1)  1 ] 


= 0.9717. 

6)  The  aean-time-to-failure  is 

1 1 

K (N  - n)  0.0287  (23  - n) 

If  n = 5 then 

1 1 1 

HTTP  = 1.94  lays. 

0.0287  (23-5)  0.0287(18)  0.5166 


Es-tablishinq  a Stopping  Point  for  Testing 

A cut  off  rule  for  determininq  when  to  stop  testinq  is 
simply  when  the  reliability  of  the  proqram  reaches  a desir- 
aole  reliability  for  a specific  time  period.  Let  one  assume 
that  it  is  necessary  for  proqraa  4 to  operate  1 day  with  a 
reliability  of  0.9,  When  should  one  stop  testinq  the  pro- 
qraa? The  answer  is  after  n errors  have  been  removed.  Cal- 
culations for  deteraininq  this  number  are  shown  below: 


! 
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ar  t(i)  1 = expf-0.0287  (23-n)  t (i)  ] 

0.9  = expf -0.  0287  (23-n)  (1)  ] 

0.9  = expr-0.6601  ♦ 0.0287n] 

In  0.9  = f -0. 6601  ♦ 0.0287nl 
-0.105=  -0.6601  ♦ 0. 0287n 
-0. 0287n  = -0.6601  ♦ 0.105 
-0. Q287n  = -0.5551 
n = 19.34 

n = 20,  sine*  a is  an  inteqer. 

Since  3 = 0.9175  for  n = 20,  one  should  stop  testing  the 
proqram  after  20  errors  have  been  removed. 

The  next  question  that  one  naturally  asks  is,  "approxi- 
mately how  long  will  it  take  to  remove  20  errors?"  Assuming 
systematic  testinq  procedures  are  used,  a rough  estimate  is 
calculated  by  multiplying  the  MTTF  by  the  number  of 
remaining  errors  in  the  program. 

If  5 errors  have  already  been  removed  then 
MTTF  ( 2 0-5)  = 1.94  (15)  = 29.  1 days. 

This  estimate  should  be  updated  as  errors  are  removed  from 


the  program 


